[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:40:41 dist-test-slave-bnc1 krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:40:41 dist-test-slave-bnc1 krb5kdc[399](info): set up 2 sockets
Sep 27 12:40:41 dist-test-slave-bnc1 krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260927 12:40:42.459800 392 mini_kdc.cc:121] Time spent starting KDC: real 1.581s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/krb5kdc/test-user.keytab.
Sep 27 12:40:42 dist-test-slave-bnc1 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:40:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:40:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:40:42.582875 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37707
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37707
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:40:42.692323 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:40:42.692575 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:40:42.692615 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:40:42.696604 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:40:42.696655 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:40:42.696674 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:40:42.696691 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:40:42.696705 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:40:42.701786 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37707
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:40:42.702737 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:40:42.703689 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 12:40:42.712371 412 server_base.cc:1034] running on GCE node
W20260927 12:40:42.713985 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:40:42.713943 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:40:42.714429 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:40:42.714953 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:40:42.716014 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:40:42.717192 412 hybrid_clock.cc:648] HybridClock initialized: now 1790512842717154 us; error 51 us; skew 500 ppm
Sep 27 12:40:42 dist-test-slave-bnc1 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512842, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:40:42.719802 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:40:42.720672 412 webserver.cc:460] Webserver started at http://127.0.98.62:35573/ using document root <none> and password file <none>
I20260927 12:40:42.721202 412 fs_manager.cc:359] Metadata directory not provided
I20260927 12:40:42.721289 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:40:42.721521 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:40:42.723503 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/minicluster-data/master-0/data/instance:
uuid: "f5b6ca024a0341c0a2dbc2e28d44af34"
format_stamp: "Formatted at 2026-09-27 12:40:42 on dist-test-slave-bnc1"
server_key: "be05d99e2e5913dbd10e9b46e28223dd"
server_key_iv: "5d86d3147baa6555949167b849d3d4a5"
server_key_version: "encryptionkey@0"
I20260927 12:40:42.724058 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/minicluster-data/master-0/wal/instance:
uuid: "f5b6ca024a0341c0a2dbc2e28d44af34"
format_stamp: "Formatted at 2026-09-27 12:40:42 on dist-test-slave-bnc1"
server_key: "be05d99e2e5913dbd10e9b46e28223dd"
server_key_iv: "5d86d3147baa6555949167b849d3d4a5"
server_key_version: "encryptionkey@0"
I20260927 12:40:42.727834 412 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 12:40:42.730155 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:40:42.731382 412 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 12:40:42.731518 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/minicluster-data/master-0/wal
uuid: "f5b6ca024a0341c0a2dbc2e28d44af34"
format_stamp: "Formatted at 2026-09-27 12:40:42 on dist-test-slave-bnc1"
server_key: "be05d99e2e5913dbd10e9b46e28223dd"
server_key_iv: "5d86d3147baa6555949167b849d3d4a5"
server_key_version: "encryptionkey@0"
I20260927 12:40:42.731624 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:40:42.742774 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:40:42.745427 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:40:42.745645 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:40:42.752331 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37707
I20260927 12:40:42.752358 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37707 every 8 connection(s)
I20260927 12:40:42.753232 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:40:42.755852 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:40:42.759248 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 412
I20260927 12:40:42.759366 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/minicluster-data/master-0/wal/instance
I20260927 12:40:42.760622 392 external_mini_cluster.cc:1428] Setting key 942ff3b4047339f1fb24b16cc8a809f7
I20260927 12:40:42.761430 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5b6ca024a0341c0a2dbc2e28d44af34: Bootstrap starting.
I20260927 12:40:42.764394 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f5b6ca024a0341c0a2dbc2e28d44af34: Neither blocks nor log segments found. Creating new log.
I20260927 12:40:42.765308 480 log.cc:824] T 00000000000000000000000000000000 P f5b6ca024a0341c0a2dbc2e28d44af34: Log is configured to *not* fsync() on all Append() calls
I20260927 12:40:42.767186 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5b6ca024a0341c0a2dbc2e28d44af34: No bootstrap required, opened a new log
I20260927 12:40:42.771618 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5b6ca024a0341c0a2dbc2e28d44af34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5b6ca024a0341c0a2dbc2e28d44af34" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37707 } }
I20260927 12:40:42.771946 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P f5b6ca024a0341c0a2dbc2e28d44af34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:40:42.772014 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5b6ca024a0341c0a2dbc2e28d44af34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5b6ca024a0341c0a2dbc2e28d44af34, State: Initialized, Role: FOLLOWER
I20260927 12:40:42.772590 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5b6ca024a0341c0a2dbc2e28d44af34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5b6ca024a0341c0a2dbc2e28d44af34" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37707 } }
I20260927 12:40:42.772786 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P f5b6ca024a0341c0a2dbc2e28d44af34 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:40:42.772897 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P f5b6ca024a0341c0a2dbc2e28d44af34 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:40:42.773062 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5b6ca024a0341c0a2dbc2e28d44af34 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:40:42.774354 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P f5b6ca024a0341c0a2dbc2e28d44af34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5b6ca024a0341c0a2dbc2e28d44af34" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37707 } }
I20260927 12:40:42.774741 480 leader_election.cc:302] T 00000000000000000000000000000000 P f5b6ca024a0341c0a2dbc2e28d44af34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f5b6ca024a0341c0a2dbc2e28d44af34; no voters:
I20260927 12:40:42.775059 480 leader_election.cc:288] T 00000000000000000000000000000000 P f5b6ca024a0341c0a2dbc2e28d44af34 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:40:42.775193 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f5b6ca024a0341c0a2dbc2e28d44af34 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:40:42.775466 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P f5b6ca024a0341c0a2dbc2e28d44af34 [term 1 LEADER]: Becoming Leader. State: Replica: f5b6ca024a0341c0a2dbc2e28d44af34, State: Running, Role: LEADER
I20260927 12:40:42.775849 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5b6ca024a0341c0a2dbc2e28d44af34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5b6ca024a0341c0a2dbc2e28d44af34" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37707 } }
I20260927 12:40:42.776537 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5b6ca024a0341c0a2dbc2e28d44af34 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:40:42.777619 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5b6ca024a0341c0a2dbc2e28d44af34 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f5b6ca024a0341c0a2dbc2e28d44af34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5b6ca024a0341c0a2dbc2e28d44af34" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37707 } } }
I20260927 12:40:42.777781 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5b6ca024a0341c0a2dbc2e28d44af34 [sys.catalog]: This master's current role is: LEADER
I20260927 12:40:42.777762 487 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5b6ca024a0341c0a2dbc2e28d44af34 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5b6ca024a0341c0a2dbc2e28d44af34. Latest consensus state: current_term: 1 leader_uuid: "f5b6ca024a0341c0a2dbc2e28d44af34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5b6ca024a0341c0a2dbc2e28d44af34" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37707 } } }
I20260927 12:40:42.777832 487 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5b6ca024a0341c0a2dbc2e28d44af34 [sys.catalog]: This master's current role is: LEADER
I20260927 12:40:42.778302 494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 27 12:40:42 dist-test-slave-bnc1 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:40:42.781889 494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:40:42.788002 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:42.763731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37046 (local address 127.0.98.62:37707)
0927 12:40:42.764145 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:42.764160 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:42.765255 (+ 1095us) server_negotiation.cc:407] Connection header received
0927 12:40:42.768379 (+ 3124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:42.768395 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:42.768675 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:42.768988 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:40:42.771812 (+ 2824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:42.773425 (+ 1613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:42.775225 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:42.776009 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:42.776172 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:42.782087 (+ 5915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:40:42.782094 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:40:42.782110 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:40:42.782145 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:40:42.784442 (+ 2297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:42.785044 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:42.785051 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:42.785060 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:42.785192 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:42.785609 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:42.785612 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:42.785614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:42.786023 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:40:42.786266 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:42.786858 (+ 592us) server_negotiation.cc:299] Negotiation successful
0927 12:40:42.787132 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":132,"threads_started":1}
I20260927 12:40:42.790453 494 catalog_manager.cc:1347] Generated new cluster ID: 38ce81b65e6d4731b27af7c44c699cf4
I20260927 12:40:42.790516 494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:40:42.811208 494 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:40:42.812450 494 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:40:42.816402 494 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f5b6ca024a0341c0a2dbc2e28d44af34: Generated new TSK 0
I20260927 12:40:42.817214 494 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 12:40:42.848336 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37707
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:40:42.965051 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:40:42.965327 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:40:42.965376 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:40:42.969501 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:40:42.969604 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:40:42.969762 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:40:42.974875 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37707
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:40:42.975905 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:40:42.977011 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:40:42.986820 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:40:42.986820 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:40:42.987021 506 server_base.cc:1034] running on GCE node
W20260927 12:40:42.986837 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:40:42.987521 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:40:42.988117 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:40:42.989269 506 hybrid_clock.cc:648] HybridClock initialized: now 1790512842989234 us; error 49 us; skew 500 ppm
Sep 27 12:40:42 dist-test-slave-bnc1 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512842, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:40:42.991880 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 12:40:42.992763 506 webserver.cc:460] Webserver started at http://127.0.98.1:39261/ using document root <none> and password file <none>
I20260927 12:40:42.993301 506 fs_manager.cc:359] Metadata directory not provided
I20260927 12:40:42.993410 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:40:42.993685 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:40:42.995766 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/minicluster-data/ts-0/data/instance:
uuid: "5416b80a02714c81a7938625afd0ba1a"
format_stamp: "Formatted at 2026-09-27 12:40:42 on dist-test-slave-bnc1"
server_key: "3d100361783560b6afea801541e15157"
server_key_iv: "2c4412aaa686dada7f312643a805657b"
server_key_version: "encryptionkey@0"
I20260927 12:40:42.996320 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5416b80a02714c81a7938625afd0ba1a"
format_stamp: "Formatted at 2026-09-27 12:40:42 on dist-test-slave-bnc1"
server_key: "3d100361783560b6afea801541e15157"
server_key_iv: "2c4412aaa686dada7f312643a805657b"
server_key_version: "encryptionkey@0"
I20260927 12:40:42.999890 506 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 12:40:43.002249 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:40:43.003479 506 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:40:43.003611 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/minicluster-data/ts-0/wal
uuid: "5416b80a02714c81a7938625afd0ba1a"
format_stamp: "Formatted at 2026-09-27 12:40:42 on dist-test-slave-bnc1"
server_key: "3d100361783560b6afea801541e15157"
server_key_iv: "2c4412aaa686dada7f312643a805657b"
server_key_version: "encryptionkey@0"
I20260927 12:40:43.003724 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:40:43.014514 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:40:43.017154 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:40:43.017381 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:40:43.017984 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:40:43.018726 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:40:43.018793 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:40:43.018857 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:40:43.018904 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:40:43.027947 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46849
I20260927 12:40:43.027968 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46849 every 8 connection(s)
I20260927 12:40:43.028797 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:40:43.034493 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 506
I20260927 12:40:43.034611 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/minicluster-data/ts-0/wal/instance
I20260927 12:40:43.034905 392 external_mini_cluster.cc:1428] Setting key 173a294b521f4a9c85c0aa3f6bcb7b7d
Sep 27 12:40:43 dist-test-slave-bnc1 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512842, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:40:43.043336 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:43.030699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34057 (local address 127.0.98.62:37707)
0927 12:40:43.030820 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:43.030824 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:43.031686 (+ 862us) server_negotiation.cc:407] Connection header received
0927 12:40:43.033036 (+ 1350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:43.033039 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:43.033091 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:43.033170 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:40:43.034451 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:43.035124 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:43.036483 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:43.036731 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:43.036850 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:43.039335 (+ 2485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:40:43.039342 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:40:43.039348 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:40:43.039380 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:40:43.041190 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:43.041728 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:43.041733 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:43.041737 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:43.041866 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:43.042200 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:43.042205 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:43.042207 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:43.042435 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:40:43.042547 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:43.043010 (+ 463us) server_negotiation.cc:299] Negotiation successful
0927 12:40:43.043149 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 12:40:43.044042 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:43.031095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37707 (local address 127.0.98.1:34057)
0927 12:40:43.031531 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0927 12:40:43.031571 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 12:40:43.032819 (+ 1248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:40:43.033360 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:40:43.033368 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:40:43.033770 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:40:43.034220 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:43.034233 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:43.035246 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:43.035250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:43.036369 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:43.036378 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:43.036882 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:43.036886 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:43.037049 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:43.037719 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:40:43.037745 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:40:43.039170 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:40:43.041315 (+ 2145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:40:43.041319 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:40:43.041330 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:40:43.041631 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:40:43.041965 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:40:43.041967 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:40:43.041968 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:40:43.042093 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:40:43.042540 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:40:43.042544 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:40:43.042758 (+ 214us) client_negotiation.cc:769] Sending connection context
0927 12:40:43.042904 (+ 146us) client_negotiation.cc:240] Negotiation successful
0927 12:40:43.043141 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":123,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 12:40:43.045214 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37707
I20260927 12:40:43.045424 635 heartbeater.cc:464] Registering TS with master...
I20260927 12:40:43.045915 635 heartbeater.cc:511] Master 127.0.98.62:37707 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 12:40:43.047639 444 ts_manager.cc:194] Registered new tserver with Master: 5416b80a02714c81a7938625afd0ba1a (127.0.98.1:46849)
I20260927 12:40:43.048980 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34057
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 12:40:43.061230 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37707
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:40:43.170315 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:40:43.170595 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:40:43.170686 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:40:43.174645 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:40:43.174719 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:40:43.174824 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:40:43.180020 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37707
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:40:43.180984 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:40:43.182014 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:40:43.191361 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:40:43.191386 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:40:43.191368 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:40:43.191588 640 server_base.cc:1034] running on GCE node
I20260927 12:40:43.192071 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:40:43.192728 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:40:43.193928 640 hybrid_clock.cc:648] HybridClock initialized: now 1790512843193907 us; error 29 us; skew 500 ppm
Sep 27 12:40:43 dist-test-slave-bnc1 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512843, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:40:43.196800 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 12:40:43.197751 640 webserver.cc:460] Webserver started at http://127.0.98.2:34731/ using document root <none> and password file <none>
I20260927 12:40:43.198295 640 fs_manager.cc:359] Metadata directory not provided
I20260927 12:40:43.198387 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:40:43.198626 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:40:43.200616 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/minicluster-data/ts-1/data/instance:
uuid: "163e16c197184f05b0e204991caa0500"
format_stamp: "Formatted at 2026-09-27 12:40:43 on dist-test-slave-bnc1"
server_key: "49e565dca1fdd8ff1dacc94a33e2e84a"
server_key_iv: "4c2e25e0968080aa999d5c613add53d8"
server_key_version: "encryptionkey@0"
I20260927 12:40:43.201221 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/minicluster-data/ts-1/wal/instance:
uuid: "163e16c197184f05b0e204991caa0500"
format_stamp: "Formatted at 2026-09-27 12:40:43 on dist-test-slave-bnc1"
server_key: "49e565dca1fdd8ff1dacc94a33e2e84a"
server_key_iv: "4c2e25e0968080aa999d5c613add53d8"
server_key_version: "encryptionkey@0"
I20260927 12:40:43.204942 640 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 12:40:43.207645 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:40:43.208891 640 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 12:40:43.209028 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/minicluster-data/ts-1/wal
uuid: "163e16c197184f05b0e204991caa0500"
format_stamp: "Formatted at 2026-09-27 12:40:43 on dist-test-slave-bnc1"
server_key: "49e565dca1fdd8ff1dacc94a33e2e84a"
server_key_iv: "4c2e25e0968080aa999d5c613add53d8"
server_key_version: "encryptionkey@0"
I20260927 12:40:43.209141 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:40:43.231568 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:40:43.235466 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:40:43.235666 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:40:43.236243 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:40:43.237051 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:40:43.237126 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:40:43.237197 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:40:43.237233 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:40:43.246325 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43713
I20260927 12:40:43.246361 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43713 every 8 connection(s)
I20260927 12:40:43.247155 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 12:40:43 dist-test-slave-bnc1 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512843, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:40:43.256938 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 640
I20260927 12:40:43.257063 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/minicluster-data/ts-1/wal/instance
I20260927 12:40:43.257395 392 external_mini_cluster.cc:1428] Setting key 63cf4ff68bd7f2d53786e36019c8c260
I20260927 12:40:43.260006 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:43.248897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34813 (local address 127.0.98.62:37707)
0927 12:40:43.249059 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:43.249065 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:43.249962 (+ 897us) server_negotiation.cc:407] Connection header received
0927 12:40:43.251192 (+ 1230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:43.251194 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:43.251237 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:43.251318 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:40:43.252365 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:43.252879 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:43.253989 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:43.254203 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:43.254315 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:43.256371 (+ 2056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:40:43.256374 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:40:43.256376 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:40:43.256398 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:40:43.257914 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:43.258488 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:43.258490 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:43.258491 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:43.258569 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:43.258960 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:43.258963 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:43.258965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:43.259138 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:40:43.259238 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:43.259728 (+ 490us) server_negotiation.cc:299] Negotiation successful
0927 12:40:43.259845 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 12:40:43.260823 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:43.249375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37707 (local address 127.0.98.2:34813)
0927 12:40:43.249812 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0927 12:40:43.249861 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 12:40:43.251007 (+ 1146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:40:43.251470 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:40:43.251476 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:40:43.251790 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:40:43.252245 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:43.252257 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:43.252995 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:43.252997 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:43.253887 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:43.253895 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:43.254343 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:43.254346 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:43.254467 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:43.254946 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:40:43.254960 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:40:43.256230 (+ 1270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:40:43.258033 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:40:43.258038 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:40:43.258052 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:40:43.258385 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:40:43.258683 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:40:43.258686 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:40:43.258689 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:40:43.258873 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:40:43.259245 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:40:43.259251 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:40:43.259477 (+ 226us) client_negotiation.cc:769] Sending connection context
0927 12:40:43.259673 (+ 196us) client_negotiation.cc:240] Negotiation successful
0927 12:40:43.259978 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":172,"threads_started":1}
I20260927 12:40:43.261922 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37707
I20260927 12:40:43.262162 769 heartbeater.cc:464] Registering TS with master...
I20260927 12:40:43.262737 769 heartbeater.cc:511] Master 127.0.98.62:37707 requested a full tablet report, sending...
I20260927 12:40:43.263871 444 ts_manager.cc:194] Registered new tserver with Master: 163e16c197184f05b0e204991caa0500 (127.0.98.2:43713)
I20260927 12:40:43.264473 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34813
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 12:40:43.284720 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37707
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:40:43.395011 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:40:43.395289 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:40:43.395367 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:40:43.399423 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:40:43.399474 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:40:43.399578 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:40:43.404759 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37707
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:40:43.405851 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:40:43.406889 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:40:43.416827 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:40:43.416874 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:40:43.416834 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:40:43.416869 774 server_base.cc:1034] running on GCE node
I20260927 12:40:43.417392 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:40:43.418031 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:40:43.419180 774 hybrid_clock.cc:648] HybridClock initialized: now 1790512843419151 us; error 45 us; skew 500 ppm
Sep 27 12:40:43 dist-test-slave-bnc1 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512843, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:40:43.421859 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 12:40:43.422780 774 webserver.cc:460] Webserver started at http://127.0.98.3:36267/ using document root <none> and password file <none>
I20260927 12:40:43.423314 774 fs_manager.cc:359] Metadata directory not provided
I20260927 12:40:43.423373 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:40:43.423630 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:40:43.425621 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/minicluster-data/ts-2/data/instance:
uuid: "9f0b7dec88f24c74ae04eaf668d67d26"
format_stamp: "Formatted at 2026-09-27 12:40:43 on dist-test-slave-bnc1"
server_key: "5fdae4cbc1fbe3f3c9b867841fe55b9e"
server_key_iv: "bdd15e1d2a141030dbb6db1ea12588d0"
server_key_version: "encryptionkey@0"
I20260927 12:40:43.426190 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9f0b7dec88f24c74ae04eaf668d67d26"
format_stamp: "Formatted at 2026-09-27 12:40:43 on dist-test-slave-bnc1"
server_key: "5fdae4cbc1fbe3f3c9b867841fe55b9e"
server_key_iv: "bdd15e1d2a141030dbb6db1ea12588d0"
server_key_version: "encryptionkey@0"
I20260927 12:40:43.429723 774 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 12:40:43.432008 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:40:43.433252 774 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:40:43.433387 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/minicluster-data/ts-2/wal
uuid: "9f0b7dec88f24c74ae04eaf668d67d26"
format_stamp: "Formatted at 2026-09-27 12:40:43 on dist-test-slave-bnc1"
server_key: "5fdae4cbc1fbe3f3c9b867841fe55b9e"
server_key_iv: "bdd15e1d2a141030dbb6db1ea12588d0"
server_key_version: "encryptionkey@0"
I20260927 12:40:43.433487 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:40:43.451529 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:40:43.454522 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:40:43.454718 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:40:43.455294 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:40:43.456081 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:40:43.456128 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:40:43.456195 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:40:43.456235 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:40:43.465468 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35519
I20260927 12:40:43.465492 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35519 every 8 connection(s)
I20260927 12:40:43.466370 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:40:43.470064 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 774
I20260927 12:40:43.470187 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790512840836255-392-0/minicluster-data/ts-2/wal/instance
I20260927 12:40:43.470448 392 external_mini_cluster.cc:1428] Setting key 75f0cee1ebd1c9d9e3924dae35cf71b4
Sep 27 12:40:43 dist-test-slave-bnc1 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512843, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:40:43.480625 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:43.468309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47735 (local address 127.0.98.62:37707)
0927 12:40:43.468450 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:43.468454 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:43.469131 (+ 677us) server_negotiation.cc:407] Connection header received
0927 12:40:43.470680 (+ 1549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:43.470684 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:43.470744 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:43.470855 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:40:43.472293 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:43.472979 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:43.474131 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:43.474367 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:43.474477 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:43.477174 (+ 2697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:40:43.477181 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:40:43.477185 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:40:43.477212 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:40:43.478703 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:43.479183 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:43.479188 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:43.479190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:43.479266 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:43.479606 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:43.479612 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:43.479613 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:43.479782 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:40:43.479889 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:43.480349 (+ 460us) server_negotiation.cc:299] Negotiation successful
0927 12:40:43.480461 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 12:40:43.481462 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:43.468537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37707 (local address 127.0.98.3:47735)
0927 12:40:43.468982 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0927 12:40:43.469022 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 12:40:43.470435 (+ 1413us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:40:43.471053 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:40:43.471063 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:40:43.471570 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:40:43.472156 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:43.472172 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:43.473096 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:43.473100 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:43.474009 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:43.474017 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:43.474507 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:43.474511 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:43.474688 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:43.475289 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:40:43.475305 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:40:43.477021 (+ 1716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:40:43.478813 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:40:43.478828 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:40:43.478841 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:40:43.479082 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:40:43.479374 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:40:43.479377 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:40:43.479379 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:40:43.479507 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:40:43.479886 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:40:43.479892 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:40:43.480119 (+ 227us) client_negotiation.cc:769] Sending connection context
0927 12:40:43.480302 (+ 183us) client_negotiation.cc:240] Negotiation successful
0927 12:40:43.480605 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":123,"threads_started":1}
I20260927 12:40:43.482511 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37707
I20260927 12:40:43.482733 903 heartbeater.cc:464] Registering TS with master...
I20260927 12:40:43.483182 903 heartbeater.cc:511] Master 127.0.98.62:37707 requested a full tablet report, sending...
I20260927 12:40:43.484220 444 ts_manager.cc:194] Registered new tserver with Master: 9f0b7dec88f24c74ae04eaf668d67d26 (127.0.98.3:35519)
I20260927 12:40:43.484697 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 12:40:43.484999 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47735
Sep 27 12:40:43 dist-test-slave-bnc1 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512843, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 12:40:43 dist-test-slave-bnc1 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512843, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 12:40:43.510514 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:43.501203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55980 (local address 127.0.98.1:46849)
0927 12:40:43.501550 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:43.501558 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:43.501624 (+ 66us) server_negotiation.cc:407] Connection header received
0927 12:40:43.501689 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:43.501711 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:43.501941 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:43.502063 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:40:43.502640 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:43.503381 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:43.504144 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:43.504537 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:43.504622 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:43.506975 (+ 2353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:40:43.506983 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:40:43.506994 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:40:43.507015 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:40:43.508832 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:43.509234 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:43.509239 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:43.509240 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:43.509317 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:43.509599 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:43.509603 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:43.509604 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:43.509846 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:40:43.509995 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:43.510156 (+ 161us) server_negotiation.cc:299] Negotiation successful
0927 12:40:43.510305 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":154,"threads_started":1}
W20260927 12:40:43.511471 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55980
Sep 27 12:40:43 dist-test-slave-bnc1 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 12:40:43 dist-test-slave-bnc1 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 12:40:43.536366 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:43.528266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55990 (local address 127.0.98.1:46849)
0927 12:40:43.528414 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:43.528418 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:43.528432 (+ 14us) server_negotiation.cc:407] Connection header received
0927 12:40:43.528557 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:43.528559 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:43.528599 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:43.528669 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:40:43.529284 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:43.529924 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:43.530687 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:43.530909 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:43.530982 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:43.533222 (+ 2240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:40:43.533225 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:40:43.533227 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:40:43.533246 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:40:43.534596 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:43.535031 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:43.535036 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:43.535037 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:43.535111 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:43.535379 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:43.535382 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:43.535384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:43.535541 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:40:43.535684 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:43.536066 (+ 382us) server_negotiation.cc:299] Negotiation successful
0927 12:40:43.536155 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 12:40:43.537412 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 506
I20260927 12:40:43.544010 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 640
I20260927 12:40:43.551403 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 774
I20260927 12:40:43.556685 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 412
2026-09-27T12:40:43Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2709 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:40:43 dist-test-slave-bnc1 krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:40:43 dist-test-slave-bnc1 krb5kdc[917](info): set up 2 sockets
Sep 27 12:40:43 dist-test-slave-bnc1 krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260927 12:40:44.589319 392 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/krb5kdc/test-user.keytab.
Sep 27 12:40:44 dist-test-slave-bnc1 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512844, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:40:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:40:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:40:44.712380 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44671
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44671
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:40:44.820454 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:40:44.820732 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:40:44.820812 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:40:44.824738 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:40:44.824836 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:40:44.824877 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:40:44.824937 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:40:44.824977 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:40:44.830673 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44671
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:40:44.831698 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:40:44.832715 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:40:44.841014 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:40:44.841046 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:40:44.841068 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:40:44.841617 930 server_base.cc:1034] running on GCE node
I20260927 12:40:44.842172 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:40:44.843302 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:40:44.844480 930 hybrid_clock.cc:648] HybridClock initialized: now 1790512844844447 us; error 46 us; skew 500 ppm
Sep 27 12:40:44 dist-test-slave-bnc1 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512844, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:40:44.847671 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:40:44.848606 930 webserver.cc:460] Webserver started at http://127.0.98.62:40419/ using document root <none> and password file <none>
I20260927 12:40:44.849182 930 fs_manager.cc:359] Metadata directory not provided
I20260927 12:40:44.849274 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:40:44.849500 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:40:44.851476 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/minicluster-data/master-0/data/instance:
uuid: "cbffb33fd5a946789d2bc4d396b3cc30"
format_stamp: "Formatted at 2026-09-27 12:40:44 on dist-test-slave-bnc1"
server_key: "61aa91343dab260e00bbc6bef5271a3e"
server_key_iv: "73a462036cdf752529ace0b2048f0a71"
server_key_version: "encryptionkey@0"
I20260927 12:40:44.852021 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/minicluster-data/master-0/wal/instance:
uuid: "cbffb33fd5a946789d2bc4d396b3cc30"
format_stamp: "Formatted at 2026-09-27 12:40:44 on dist-test-slave-bnc1"
server_key: "61aa91343dab260e00bbc6bef5271a3e"
server_key_iv: "73a462036cdf752529ace0b2048f0a71"
server_key_version: "encryptionkey@0"
I20260927 12:40:44.855579 930 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 12:40:44.857827 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:40:44.858989 930 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 12:40:44.859158 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/minicluster-data/master-0/wal
uuid: "cbffb33fd5a946789d2bc4d396b3cc30"
format_stamp: "Formatted at 2026-09-27 12:40:44 on dist-test-slave-bnc1"
server_key: "61aa91343dab260e00bbc6bef5271a3e"
server_key_iv: "73a462036cdf752529ace0b2048f0a71"
server_key_version: "encryptionkey@0"
I20260927 12:40:44.859270 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:40:44.876863 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:40:44.880218 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:40:44.880431 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:40:44.887490 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44671
I20260927 12:40:44.887506 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44671 every 8 connection(s)
I20260927 12:40:44.888478 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:40:44.891237 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:40:44.896485 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbffb33fd5a946789d2bc4d396b3cc30: Bootstrap starting.
I20260927 12:40:44.898188 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 930
I20260927 12:40:44.898300 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/minicluster-data/master-0/wal/instance
I20260927 12:40:44.898543 392 external_mini_cluster.cc:1428] Setting key 4b80bb1e17810c242a91ec94df0d3014
I20260927 12:40:44.899112 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cbffb33fd5a946789d2bc4d396b3cc30: Neither blocks nor log segments found. Creating new log.
I20260927 12:40:44.899972 998 log.cc:824] T 00000000000000000000000000000000 P cbffb33fd5a946789d2bc4d396b3cc30: Log is configured to *not* fsync() on all Append() calls
I20260927 12:40:44.901871 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbffb33fd5a946789d2bc4d396b3cc30: No bootstrap required, opened a new log
I20260927 12:40:44.905134 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P cbffb33fd5a946789d2bc4d396b3cc30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbffb33fd5a946789d2bc4d396b3cc30" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44671 } }
I20260927 12:40:44.905372 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P cbffb33fd5a946789d2bc4d396b3cc30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:40:44.905423 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P cbffb33fd5a946789d2bc4d396b3cc30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbffb33fd5a946789d2bc4d396b3cc30, State: Initialized, Role: FOLLOWER
I20260927 12:40:44.905944 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P cbffb33fd5a946789d2bc4d396b3cc30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbffb33fd5a946789d2bc4d396b3cc30" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44671 } }
I20260927 12:40:44.906131 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P cbffb33fd5a946789d2bc4d396b3cc30 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:40:44.906208 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P cbffb33fd5a946789d2bc4d396b3cc30 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:40:44.906333 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cbffb33fd5a946789d2bc4d396b3cc30 [term 0 FOLLOWER]: Advancing to term 1
Sep 27 12:40:44 dist-test-slave-bnc1 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512844, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:40:44.907290 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P cbffb33fd5a946789d2bc4d396b3cc30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbffb33fd5a946789d2bc4d396b3cc30" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44671 } }
I20260927 12:40:44.907611 998 leader_election.cc:302] T 00000000000000000000000000000000 P cbffb33fd5a946789d2bc4d396b3cc30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cbffb33fd5a946789d2bc4d396b3cc30; no voters:
I20260927 12:40:44.907934 998 leader_election.cc:288] T 00000000000000000000000000000000 P cbffb33fd5a946789d2bc4d396b3cc30 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:40:44.908033 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cbffb33fd5a946789d2bc4d396b3cc30 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:40:44.908270 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P cbffb33fd5a946789d2bc4d396b3cc30 [term 1 LEADER]: Becoming Leader. State: Replica: cbffb33fd5a946789d2bc4d396b3cc30, State: Running, Role: LEADER
I20260927 12:40:44.908586 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P cbffb33fd5a946789d2bc4d396b3cc30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbffb33fd5a946789d2bc4d396b3cc30" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44671 } }
I20260927 12:40:44.909238 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P cbffb33fd5a946789d2bc4d396b3cc30 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:40:44.910336 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbffb33fd5a946789d2bc4d396b3cc30 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cbffb33fd5a946789d2bc4d396b3cc30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbffb33fd5a946789d2bc4d396b3cc30" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44671 } } }
I20260927 12:40:44.910490 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbffb33fd5a946789d2bc4d396b3cc30 [sys.catalog]: This master's current role is: LEADER
I20260927 12:40:44.910336 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbffb33fd5a946789d2bc4d396b3cc30 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbffb33fd5a946789d2bc4d396b3cc30. Latest consensus state: current_term: 1 leader_uuid: "cbffb33fd5a946789d2bc4d396b3cc30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbffb33fd5a946789d2bc4d396b3cc30" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44671 } } }
I20260927 12:40:44.910842 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbffb33fd5a946789d2bc4d396b3cc30 [sys.catalog]: This master's current role is: LEADER
I20260927 12:40:44.911147 1012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:40:44.913314 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:44.899732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37408 (local address 127.0.98.62:44671)
0927 12:40:44.900209 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:44.900225 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:44.900259 (+ 34us) server_negotiation.cc:407] Connection header received
0927 12:40:44.901027 (+ 768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:44.901046 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:44.901368 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:44.901758 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:40:44.902500 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:44.903474 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:44.904306 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:44.904781 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:44.904952 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:44.907796 (+ 2844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:40:44.907802 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:40:44.907819 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:40:44.907855 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:40:44.909923 (+ 2068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:44.910478 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:44.910484 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:44.910489 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:44.910615 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:44.910958 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:44.910962 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:44.910967 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:44.911540 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:40:44.911752 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:44.912069 (+ 317us) server_negotiation.cc:299] Negotiation successful
0927 12:40:44.912371 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":127,"threads_started":1}
I20260927 12:40:44.914029 1012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:40:44.919255 1012 catalog_manager.cc:1347] Generated new cluster ID: a398e9d2d69447b69f23f7475364a714
I20260927 12:40:44.919318 1012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:40:44.937115 1012 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:40:44.938390 1012 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:40:44.945387 1012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cbffb33fd5a946789d2bc4d396b3cc30: Generated new TSK 0
I20260927 12:40:44.946329 1012 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 12:40:44.978698 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44671
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:40:45.090392 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:40:45.090655 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:40:45.090741 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:40:45.094837 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:40:45.094898 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:40:45.095041 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:40:45.100502 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44671
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:40:45.101598 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:40:45.102689 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:40:45.112542 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:40:45.112560 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:40:45.112751 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:40:45.112938 1024 server_base.cc:1034] running on GCE node
I20260927 12:40:45.113322 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:40:45.114013 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:40:45.115200 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790512845115185 us; error 31 us; skew 500 ppm
Sep 27 12:40:45 dist-test-slave-bnc1 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512845, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:40:45.118013 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 12:40:45.118964 1024 webserver.cc:460] Webserver started at http://127.0.98.1:43819/ using document root <none> and password file <none>
I20260927 12:40:45.119513 1024 fs_manager.cc:359] Metadata directory not provided
I20260927 12:40:45.119609 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:40:45.119839 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:40:45.121930 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/minicluster-data/ts-0/data/instance:
uuid: "024acc06f080426da5f1aa4a26397a70"
format_stamp: "Formatted at 2026-09-27 12:40:45 on dist-test-slave-bnc1"
server_key: "5666ef5835a6e134ebbd4da9499c9d2f"
server_key_iv: "56cab2b2b6c6fc60f6fae817270e49a1"
server_key_version: "encryptionkey@0"
I20260927 12:40:45.122475 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/minicluster-data/ts-0/wal/instance:
uuid: "024acc06f080426da5f1aa4a26397a70"
format_stamp: "Formatted at 2026-09-27 12:40:45 on dist-test-slave-bnc1"
server_key: "5666ef5835a6e134ebbd4da9499c9d2f"
server_key_iv: "56cab2b2b6c6fc60f6fae817270e49a1"
server_key_version: "encryptionkey@0"
I20260927 12:40:45.126088 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 12:40:45.128504 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:40:45.129803 1024 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 12:40:45.129940 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/minicluster-data/ts-0/wal
uuid: "024acc06f080426da5f1aa4a26397a70"
format_stamp: "Formatted at 2026-09-27 12:40:45 on dist-test-slave-bnc1"
server_key: "5666ef5835a6e134ebbd4da9499c9d2f"
server_key_iv: "56cab2b2b6c6fc60f6fae817270e49a1"
server_key_version: "encryptionkey@0"
I20260927 12:40:45.130059 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:40:45.141978 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:40:45.145650 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:40:45.145861 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:40:45.146507 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:40:45.147365 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:40:45.147441 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:40:45.147528 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:40:45.147585 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:40:45.156535 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38565
I20260927 12:40:45.156548 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38565 every 8 connection(s)
I20260927 12:40:45.157413 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:40:45.165392 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 1024
I20260927 12:40:45.165508 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/minicluster-data/ts-0/wal/instance
I20260927 12:40:45.165820 392 external_mini_cluster.cc:1428] Setting key 7c4cc5721f8ccb1ec197678363b6b705
Sep 27 12:40:45 dist-test-slave-bnc1 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512845, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:40:45.172277 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:45.159236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40087 (local address 127.0.98.62:44671)
0927 12:40:45.159369 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:45.159372 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:45.160147 (+ 775us) server_negotiation.cc:407] Connection header received
0927 12:40:45.161428 (+ 1281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:45.161431 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:45.161476 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:45.161581 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:40:45.162831 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:45.163497 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:45.164726 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:45.164960 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:45.165054 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:45.168073 (+ 3019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:40:45.168076 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:40:45.168101 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:40:45.168125 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:40:45.170062 (+ 1937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:45.170561 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:45.170567 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:45.170572 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:45.170669 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:45.170997 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:45.171003 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:45.171016 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:45.171285 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:40:45.171411 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:45.171961 (+ 550us) server_negotiation.cc:299] Negotiation successful
0927 12:40:45.172136 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 12:40:45.173504 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:45.159559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44671 (local address 127.0.98.1:40087)
0927 12:40:45.159994 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0927 12:40:45.160035 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 12:40:45.161225 (+ 1190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:40:45.161787 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:40:45.161795 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:40:45.162162 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:40:45.162695 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:45.162709 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:45.163661 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:45.163664 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:45.164575 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:45.164585 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:45.165139 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:45.165146 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:45.165379 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:45.166042 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:40:45.166065 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:40:45.167849 (+ 1784us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:40:45.170177 (+ 2328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:40:45.170182 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:40:45.170195 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:40:45.170459 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:40:45.170765 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:40:45.170767 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:40:45.170768 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:40:45.170903 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:40:45.171437 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:40:45.171444 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:40:45.171718 (+ 274us) client_negotiation.cc:769] Sending connection context
0927 12:40:45.171882 (+ 164us) client_negotiation.cc:240] Negotiation successful
0927 12:40:45.172238 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":112,"threads_started":1}
I20260927 12:40:45.174816 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44671
I20260927 12:40:45.175055 1153 heartbeater.cc:464] Registering TS with master...
I20260927 12:40:45.175544 1153 heartbeater.cc:511] Master 127.0.98.62:44671 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 12:40:45.177067 962 ts_manager.cc:194] Registered new tserver with Master: 024acc06f080426da5f1aa4a26397a70 (127.0.98.1:38565)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 12:40:45.178385 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40087
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 12:40:45.192548 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44671
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:40:45.303046 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:40:45.303350 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:40:45.303427 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:40:45.307565 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:40:45.307658 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:40:45.307749 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:40:45.313171 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44671
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:40:45.314320 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:40:45.315340 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:40:45.324698 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:40:45.324865 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:40:45.324918 1158 server_base.cc:1034] running on GCE node
W20260927 12:40:45.326154 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:40:45.326567 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:40:45.327243 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:40:45.328438 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790512845328400 us; error 48 us; skew 500 ppm
Sep 27 12:40:45 dist-test-slave-bnc1 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512845, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:40:45.331174 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 12:40:45.332053 1158 webserver.cc:460] Webserver started at http://127.0.98.2:37455/ using document root <none> and password file <none>
I20260927 12:40:45.332602 1158 fs_manager.cc:359] Metadata directory not provided
I20260927 12:40:45.332693 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:40:45.332971 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:40:45.335124 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/minicluster-data/ts-1/data/instance:
uuid: "81b0eab12dac4fc995d9679a4c3515bf"
format_stamp: "Formatted at 2026-09-27 12:40:45 on dist-test-slave-bnc1"
server_key: "fbcc38e6c2d877a6b79b056a41295e40"
server_key_iv: "c0220e156af99c9fc610e1c3cd45a00c"
server_key_version: "encryptionkey@0"
I20260927 12:40:45.335680 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/minicluster-data/ts-1/wal/instance:
uuid: "81b0eab12dac4fc995d9679a4c3515bf"
format_stamp: "Formatted at 2026-09-27 12:40:45 on dist-test-slave-bnc1"
server_key: "fbcc38e6c2d877a6b79b056a41295e40"
server_key_iv: "c0220e156af99c9fc610e1c3cd45a00c"
server_key_version: "encryptionkey@0"
I20260927 12:40:45.339395 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 12:40:45.341743 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:40:45.342908 1158 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 12:40:45.343050 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/minicluster-data/ts-1/wal
uuid: "81b0eab12dac4fc995d9679a4c3515bf"
format_stamp: "Formatted at 2026-09-27 12:40:45 on dist-test-slave-bnc1"
server_key: "fbcc38e6c2d877a6b79b056a41295e40"
server_key_iv: "c0220e156af99c9fc610e1c3cd45a00c"
server_key_version: "encryptionkey@0"
I20260927 12:40:45.343185 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:40:45.400880 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:40:45.403999 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:40:45.404229 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:40:45.404836 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:40:45.405714 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:40:45.405778 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:40:45.405845 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:40:45.405890 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:40:45.415113 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42433
I20260927 12:40:45.415132 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42433 every 8 connection(s)
I20260927 12:40:45.415961 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:40:45.418922 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 1158
I20260927 12:40:45.419042 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/minicluster-data/ts-1/wal/instance
I20260927 12:40:45.419305 392 external_mini_cluster.cc:1428] Setting key d1e612cce8f25d8c9db12f406b03746a
Sep 27 12:40:45 dist-test-slave-bnc1 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512845, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 12:40:45.430470 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:45.417707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49723 (local address 127.0.98.62:44671)
0927 12:40:45.417833 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:45.417838 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:45.418653 (+ 815us) server_negotiation.cc:407] Connection header received
0927 12:40:45.420243 (+ 1590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:45.420246 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:45.420284 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:45.420353 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:40:45.421535 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:45.422326 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:45.423793 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:45.424201 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:45.424323 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:45.426973 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:40:45.426978 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:40:45.426983 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:40:45.427016 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:40:45.428599 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:45.429105 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:45.429108 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:45.429109 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:45.429179 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:45.429465 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:45.429467 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:45.429468 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:45.429724 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:40:45.429802 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:45.430240 (+ 438us) server_negotiation.cc:299] Negotiation successful
0927 12:40:45.430329 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 12:40:45.431352 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:45.418095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44671 (local address 127.0.98.2:49723)
0927 12:40:45.418510 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0927 12:40:45.418551 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 12:40:45.420026 (+ 1475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:40:45.420554 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:40:45.420563 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:40:45.420938 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:40:45.421397 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:45.421409 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:45.422450 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:45.422456 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:45.423671 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:45.423681 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:45.424366 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:45.424371 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:45.424550 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:45.425135 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:40:45.425150 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:40:45.426809 (+ 1659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:40:45.428717 (+ 1908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:40:45.428722 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:40:45.428732 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:40:45.429010 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:40:45.429262 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:40:45.429264 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:40:45.429265 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:40:45.429389 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:40:45.429823 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:40:45.429827 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:40:45.430037 (+ 210us) client_negotiation.cc:769] Sending connection context
0927 12:40:45.430199 (+ 162us) client_negotiation.cc:240] Negotiation successful
0927 12:40:45.430433 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":103,"threads_started":1}
I20260927 12:40:45.432542 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44671
I20260927 12:40:45.432797 1287 heartbeater.cc:464] Registering TS with master...
I20260927 12:40:45.433326 1287 heartbeater.cc:511] Master 127.0.98.62:44671 requested a full tablet report, sending...
I20260927 12:40:45.434577 962 ts_manager.cc:194] Registered new tserver with Master: 81b0eab12dac4fc995d9679a4c3515bf (127.0.98.2:42433)
I20260927 12:40:45.435181 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49723
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 12:40:45.448251 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44671
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:40:45.557301 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:40:45.557612 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:40:45.557698 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:40:45.561681 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:40:45.561730 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:40:45.561789 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:40:45.567135 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44671
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:40:45.568221 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:40:45.569223 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:40:45.579124 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:40:45.579130 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:40:45.579212 1292 server_base.cc:1034] running on GCE node
W20260927 12:40:45.579330 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:40:45.579797 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:40:45.580430 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:40:45.581666 1292 hybrid_clock.cc:648] HybridClock initialized: now 1790512845581636 us; error 37 us; skew 500 ppm
Sep 27 12:40:45 dist-test-slave-bnc1 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512845, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:40:45.587731 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 12:40:45.588675 1292 webserver.cc:460] Webserver started at http://127.0.98.3:37533/ using document root <none> and password file <none>
I20260927 12:40:45.589200 1292 fs_manager.cc:359] Metadata directory not provided
I20260927 12:40:45.589257 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:40:45.589422 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:40:45.591512 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/minicluster-data/ts-2/data/instance:
uuid: "7da678c8cb71478298e71b56458c84a8"
format_stamp: "Formatted at 2026-09-27 12:40:45 on dist-test-slave-bnc1"
server_key: "dd526861867f1e03e068e9e6670b9bd5"
server_key_iv: "659a79e18038cb2cbe72a64df080ec03"
server_key_version: "encryptionkey@0"
I20260927 12:40:45.592008 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7da678c8cb71478298e71b56458c84a8"
format_stamp: "Formatted at 2026-09-27 12:40:45 on dist-test-slave-bnc1"
server_key: "dd526861867f1e03e068e9e6670b9bd5"
server_key_iv: "659a79e18038cb2cbe72a64df080ec03"
server_key_version: "encryptionkey@0"
I20260927 12:40:45.595683 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 12:40:45.597962 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:40:45.599104 1292 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 12:40:45.599228 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/minicluster-data/ts-2/wal
uuid: "7da678c8cb71478298e71b56458c84a8"
format_stamp: "Formatted at 2026-09-27 12:40:45 on dist-test-slave-bnc1"
server_key: "dd526861867f1e03e068e9e6670b9bd5"
server_key_iv: "659a79e18038cb2cbe72a64df080ec03"
server_key_version: "encryptionkey@0"
I20260927 12:40:45.599345 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:40:45.620093 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:40:45.622857 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:40:45.623046 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:40:45.623589 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:40:45.624389 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:40:45.624433 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:40:45.624497 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:40:45.624536 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:40:45.633091 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43215
I20260927 12:40:45.633147 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43215 every 8 connection(s)
I20260927 12:40:45.634065 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 12:40:45 dist-test-slave-bnc1 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512845, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:40:45.644337 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 1292
I20260927 12:40:45.644474 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790512840836255-392-0/minicluster-data/ts-2/wal/instance
I20260927 12:40:45.644764 392 external_mini_cluster.cc:1428] Setting key f778424bac553429ca42c3cc4d21b1ff
I20260927 12:40:45.648020 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:45.635836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46477 (local address 127.0.98.62:44671)
0927 12:40:45.635962 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:45.635965 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:45.636731 (+ 766us) server_negotiation.cc:407] Connection header received
0927 12:40:45.638116 (+ 1385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:45.638119 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:45.638174 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:45.638254 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:40:45.639337 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:45.639878 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:45.640992 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:45.641229 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:45.641341 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:45.643751 (+ 2410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:40:45.643755 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:40:45.643758 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:40:45.643791 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:40:45.645769 (+ 1978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:45.646376 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:45.646380 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:45.646382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:45.646465 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:45.646816 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:45.646819 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:45.646821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:45.647055 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:40:45.647196 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:45.647749 (+ 553us) server_negotiation.cc:299] Negotiation successful
0927 12:40:45.647868 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 12:40:45.648859 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:45.636171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44671 (local address 127.0.98.3:46477)
0927 12:40:45.636582 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0927 12:40:45.636622 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 12:40:45.637909 (+ 1287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:40:45.638413 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:40:45.638419 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:40:45.638748 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:40:45.639199 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:45.639211 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:45.640000 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:45.640004 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:45.640870 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:45.640879 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:45.641375 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:45.641379 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:45.641540 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:45.642218 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:40:45.642233 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:40:45.643587 (+ 1354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:40:45.645888 (+ 2301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:40:45.645893 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:40:45.645909 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:40:45.646286 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:40:45.646565 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:40:45.646568 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:40:45.646570 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:40:45.646712 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:40:45.647191 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:40:45.647197 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:40:45.647509 (+ 312us) client_negotiation.cc:769] Sending connection context
0927 12:40:45.647697 (+ 188us) client_negotiation.cc:240] Negotiation successful
0927 12:40:45.648021 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":106,"threads_started":1}
I20260927 12:40:45.649957 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44671
I20260927 12:40:45.650183 1421 heartbeater.cc:464] Registering TS with master...
I20260927 12:40:45.650621 1421 heartbeater.cc:511] Master 127.0.98.62:44671 requested a full tablet report, sending...
I20260927 12:40:45.651553 962 ts_manager.cc:194] Registered new tserver with Master: 7da678c8cb71478298e71b56458c84a8 (127.0.98.3:43215)
I20260927 12:40:45.652076 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46477
I20260927 12:40:45.658758 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 12:40:45.669986 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:45.662234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37416 (local address 127.0.98.62:44671)
0927 12:40:45.662396 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:45.662401 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:45.662523 (+ 122us) server_negotiation.cc:407] Connection header received
0927 12:40:45.662690 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:45.662693 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:45.662740 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:45.662829 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:40:45.663608 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:45.664130 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:45.664950 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:45.665180 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:45.665292 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:45.666818 (+ 1526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:40:45.666821 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:40:45.666823 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:40:45.666846 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:40:45.668198 (+ 1352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:45.668589 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:45.668592 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:45.668593 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:45.668666 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:45.668899 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:45.668901 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:45.668902 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:45.669147 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:40:45.669259 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:45.669657 (+ 398us) server_negotiation.cc:299] Negotiation successful
0927 12:40:45.669784 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 12:40:45.675232 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37416:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 12:40:45.678236 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:40:45.692533 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:45.686570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57178 (local address 127.0.98.2:42433)
0927 12:40:45.686978 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:45.686989 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:45.687745 (+ 756us) server_negotiation.cc:407] Connection header received
0927 12:40:45.687836 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:45.687841 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:45.688071 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:45.688246 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:40:45.688612 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:45.689851 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:45.691213 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:45.691783 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:45.691913 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:45.692082 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:45.692237 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 12:40:45.692350 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":125,"threads_started":1}
I20260927 12:40:45.692541 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:45.686442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42433 (local address 127.0.0.1:57178)
0927 12:40:45.686859 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0927 12:40:45.686906 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 12:40:45.687027 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:40:45.688233 (+ 1206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:40:45.688239 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:40:45.688260 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:40:45.688484 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:45.688492 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:45.689989 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:45.689992 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:45.691088 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:45.691097 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:45.691911 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:45.691914 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:45.692020 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:45.692054 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 12:40:45.692204 (+ 150us) client_negotiation.cc:240] Negotiation successful
0927 12:40:45.692377 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"mutex_wait_us":73,"thread_start_us":97,"threads_started":1}
I20260927 12:40:45.692806 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:45.687224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43215 (local address 127.0.0.1:48972)
0927 12:40:45.687462 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0927 12:40:45.687474 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 12:40:45.687554 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:40:45.688435 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:40:45.688437 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:40:45.688452 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:40:45.688535 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:45.688541 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:45.690208 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:45.690211 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:45.691236 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:45.691246 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:45.692448 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:45.692452 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:45.692531 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:45.692552 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 12:40:45.692617 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 12:40:45.692677 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":3,"thread_start_us":93,"threads_started":1}
I20260927 12:40:45.692544 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:45.686516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38565 (local address 127.0.0.1:33092)
0927 12:40:45.687139 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0927 12:40:45.687153 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:40:45.687241 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:40:45.688265 (+ 1024us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:40:45.688268 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:40:45.688286 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:40:45.688486 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:45.688492 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:45.689603 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:45.689609 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:45.690975 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:45.690985 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:45.692051 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:45.692055 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:45.692145 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:45.692170 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 12:40:45.692221 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 12:40:45.692372 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"mutex_wait_us":52,"thread_start_us":98,"threads_started":1}
I20260927 12:40:45.693059 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:45.687385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33092 (local address 127.0.98.1:38565)
0927 12:40:45.687801 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:45.687808 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:45.687831 (+ 23us) server_negotiation.cc:407] Connection header received
0927 12:40:45.687907 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:45.687913 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:45.688104 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:45.688454 (+ 350us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:40:45.688591 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:45.689460 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:45.691119 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:45.691910 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:45.692622 (+ 712us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:45.692782 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:45.692834 (+ 52us) server_negotiation.cc:299] Negotiation successful
0927 12:40:45.692928 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":115,"threads_started":1}
I20260927 12:40:45.693271 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:45.687529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48972 (local address 127.0.98.3:43215)
0927 12:40:45.687905 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:45.687915 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:45.687934 (+ 19us) server_negotiation.cc:407] Connection header received
0927 12:40:45.687993 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:45.687996 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:45.688149 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:45.688285 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:40:45.688638 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:45.689839 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:45.691372 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:45.692146 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:45.692256 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:45.692392 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:45.692971 (+ 579us) server_negotiation.cc:299] Negotiation successful
0927 12:40:45.693068 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":119,"threads_started":1}
I20260927 12:40:45.696094 1221 tablet_service.cc:1467] Processing CreateTablet for tablet c53ebf19f5c740a49fce5f1b88ba8b7c (DEFAULT_TABLE table=test-table [id=aa087111102b4af89741c654112e41bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:40:45.696094 1087 tablet_service.cc:1467] Processing CreateTablet for tablet c53ebf19f5c740a49fce5f1b88ba8b7c (DEFAULT_TABLE table=test-table [id=aa087111102b4af89741c654112e41bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:40:45.696099 1355 tablet_service.cc:1467] Processing CreateTablet for tablet c53ebf19f5c740a49fce5f1b88ba8b7c (DEFAULT_TABLE table=test-table [id=aa087111102b4af89741c654112e41bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:40:45.697124 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c53ebf19f5c740a49fce5f1b88ba8b7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:40:45.697131 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c53ebf19f5c740a49fce5f1b88ba8b7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:40:45.697131 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c53ebf19f5c740a49fce5f1b88ba8b7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:40:45.703612 1438 tablet_bootstrap.cc:492] T c53ebf19f5c740a49fce5f1b88ba8b7c P 81b0eab12dac4fc995d9679a4c3515bf: Bootstrap starting.
I20260927 12:40:45.704950 1440 tablet_bootstrap.cc:492] T c53ebf19f5c740a49fce5f1b88ba8b7c P 7da678c8cb71478298e71b56458c84a8: Bootstrap starting.
I20260927 12:40:45.706077 1438 tablet_bootstrap.cc:654] T c53ebf19f5c740a49fce5f1b88ba8b7c P 81b0eab12dac4fc995d9679a4c3515bf: Neither blocks nor log segments found. Creating new log.
I20260927 12:40:45.706808 1440 tablet_bootstrap.cc:654] T c53ebf19f5c740a49fce5f1b88ba8b7c P 7da678c8cb71478298e71b56458c84a8: Neither blocks nor log segments found. Creating new log.
I20260927 12:40:45.707208 1438 log.cc:824] T c53ebf19f5c740a49fce5f1b88ba8b7c P 81b0eab12dac4fc995d9679a4c3515bf: Log is configured to *not* fsync() on all Append() calls
I20260927 12:40:45.707870 1439 tablet_bootstrap.cc:492] T c53ebf19f5c740a49fce5f1b88ba8b7c P 024acc06f080426da5f1aa4a26397a70: Bootstrap starting.
I20260927 12:40:45.707996 1440 log.cc:824] T c53ebf19f5c740a49fce5f1b88ba8b7c P 7da678c8cb71478298e71b56458c84a8: Log is configured to *not* fsync() on all Append() calls
I20260927 12:40:45.709286 1438 tablet_bootstrap.cc:492] T c53ebf19f5c740a49fce5f1b88ba8b7c P 81b0eab12dac4fc995d9679a4c3515bf: No bootstrap required, opened a new log
I20260927 12:40:45.709551 1438 ts_tablet_manager.cc:1397] T c53ebf19f5c740a49fce5f1b88ba8b7c P 81b0eab12dac4fc995d9679a4c3515bf: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 12:40:45.710068 1440 tablet_bootstrap.cc:492] T c53ebf19f5c740a49fce5f1b88ba8b7c P 7da678c8cb71478298e71b56458c84a8: No bootstrap required, opened a new log
I20260927 12:40:45.710116 1439 tablet_bootstrap.cc:654] T c53ebf19f5c740a49fce5f1b88ba8b7c P 024acc06f080426da5f1aa4a26397a70: Neither blocks nor log segments found. Creating new log.
I20260927 12:40:45.710251 1440 ts_tablet_manager.cc:1397] T c53ebf19f5c740a49fce5f1b88ba8b7c P 7da678c8cb71478298e71b56458c84a8: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 12:40:45.710969 1439 log.cc:824] T c53ebf19f5c740a49fce5f1b88ba8b7c P 024acc06f080426da5f1aa4a26397a70: Log is configured to *not* fsync() on all Append() calls
I20260927 12:40:45.713533 1439 tablet_bootstrap.cc:492] T c53ebf19f5c740a49fce5f1b88ba8b7c P 024acc06f080426da5f1aa4a26397a70: No bootstrap required, opened a new log
I20260927 12:40:45.713461 1438 raft_consensus.cc:348] T c53ebf19f5c740a49fce5f1b88ba8b7c P 81b0eab12dac4fc995d9679a4c3515bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81b0eab12dac4fc995d9679a4c3515bf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42433 } } peers { permanent_uuid: "024acc06f080426da5f1aa4a26397a70" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38565 } } peers { permanent_uuid: "7da678c8cb71478298e71b56458c84a8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43215 } }
I20260927 12:40:45.713739 1439 ts_tablet_manager.cc:1397] T c53ebf19f5c740a49fce5f1b88ba8b7c P 024acc06f080426da5f1aa4a26397a70: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260927 12:40:45.713809 1438 raft_consensus.cc:374] T c53ebf19f5c740a49fce5f1b88ba8b7c P 81b0eab12dac4fc995d9679a4c3515bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:40:45.713883 1438 raft_consensus.cc:729] T c53ebf19f5c740a49fce5f1b88ba8b7c P 81b0eab12dac4fc995d9679a4c3515bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81b0eab12dac4fc995d9679a4c3515bf, State: Initialized, Role: FOLLOWER
I20260927 12:40:45.713842 1440 raft_consensus.cc:348] T c53ebf19f5c740a49fce5f1b88ba8b7c P 7da678c8cb71478298e71b56458c84a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81b0eab12dac4fc995d9679a4c3515bf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42433 } } peers { permanent_uuid: "024acc06f080426da5f1aa4a26397a70" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38565 } } peers { permanent_uuid: "7da678c8cb71478298e71b56458c84a8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43215 } }
I20260927 12:40:45.714110 1440 raft_consensus.cc:374] T c53ebf19f5c740a49fce5f1b88ba8b7c P 7da678c8cb71478298e71b56458c84a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:40:45.714161 1440 raft_consensus.cc:729] T c53ebf19f5c740a49fce5f1b88ba8b7c P 7da678c8cb71478298e71b56458c84a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7da678c8cb71478298e71b56458c84a8, State: Initialized, Role: FOLLOWER
I20260927 12:40:45.714361 1438 consensus_queue.cc:260] T c53ebf19f5c740a49fce5f1b88ba8b7c P 81b0eab12dac4fc995d9679a4c3515bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81b0eab12dac4fc995d9679a4c3515bf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42433 } } peers { permanent_uuid: "024acc06f080426da5f1aa4a26397a70" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38565 } } peers { permanent_uuid: "7da678c8cb71478298e71b56458c84a8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43215 } }
I20260927 12:40:45.714597 1440 consensus_queue.cc:260] T c53ebf19f5c740a49fce5f1b88ba8b7c P 7da678c8cb71478298e71b56458c84a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81b0eab12dac4fc995d9679a4c3515bf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42433 } } peers { permanent_uuid: "024acc06f080426da5f1aa4a26397a70" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38565 } } peers { permanent_uuid: "7da678c8cb71478298e71b56458c84a8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43215 } }
I20260927 12:40:45.715420 1438 ts_tablet_manager.cc:1428] T c53ebf19f5c740a49fce5f1b88ba8b7c P 81b0eab12dac4fc995d9679a4c3515bf: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20260927 12:40:45.715667 1440 ts_tablet_manager.cc:1428] T c53ebf19f5c740a49fce5f1b88ba8b7c P 7da678c8cb71478298e71b56458c84a8: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 12:40:45.715824 1287 heartbeater.cc:503] Master 127.0.98.62:44671 was elected leader, sending a full tablet report...
I20260927 12:40:45.716740 1421 heartbeater.cc:503] Master 127.0.98.62:44671 was elected leader, sending a full tablet report...
I20260927 12:40:45.717561 1439 raft_consensus.cc:348] T c53ebf19f5c740a49fce5f1b88ba8b7c P 024acc06f080426da5f1aa4a26397a70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81b0eab12dac4fc995d9679a4c3515bf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42433 } } peers { permanent_uuid: "024acc06f080426da5f1aa4a26397a70" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38565 } } peers { permanent_uuid: "7da678c8cb71478298e71b56458c84a8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43215 } }
I20260927 12:40:45.717825 1439 raft_consensus.cc:374] T c53ebf19f5c740a49fce5f1b88ba8b7c P 024acc06f080426da5f1aa4a26397a70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:40:45.717923 1439 raft_consensus.cc:729] T c53ebf19f5c740a49fce5f1b88ba8b7c P 024acc06f080426da5f1aa4a26397a70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 024acc06f080426da5f1aa4a26397a70, State: Initialized, Role: FOLLOWER
I20260927 12:40:45.718360 1439 consensus_queue.cc:260] T c53ebf19f5c740a49fce5f1b88ba8b7c P 024acc06f080426da5f1aa4a26397a70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81b0eab12dac4fc995d9679a4c3515bf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42433 } } peers { permanent_uuid: "024acc06f080426da5f1aa4a26397a70" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38565 } } peers { permanent_uuid: "7da678c8cb71478298e71b56458c84a8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43215 } }
I20260927 12:40:45.719098 1153 heartbeater.cc:503] Master 127.0.98.62:44671 was elected leader, sending a full tablet report...
I20260927 12:40:45.719281 1439 ts_tablet_manager.cc:1428] T c53ebf19f5c740a49fce5f1b88ba8b7c P 024acc06f080426da5f1aa4a26397a70: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
I20260927 12:40:45.823751 1445 raft_consensus.cc:482] T c53ebf19f5c740a49fce5f1b88ba8b7c P 7da678c8cb71478298e71b56458c84a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:40:45.823954 1445 raft_consensus.cc:504] T c53ebf19f5c740a49fce5f1b88ba8b7c P 7da678c8cb71478298e71b56458c84a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81b0eab12dac4fc995d9679a4c3515bf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42433 } } peers { permanent_uuid: "024acc06f080426da5f1aa4a26397a70" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38565 } } peers { permanent_uuid: "7da678c8cb71478298e71b56458c84a8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43215 } }
I20260927 12:40:45.825018 1445 leader_election.cc:288] T c53ebf19f5c740a49fce5f1b88ba8b7c P 7da678c8cb71478298e71b56458c84a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 81b0eab12dac4fc995d9679a4c3515bf (127.0.98.2:42433), 024acc06f080426da5f1aa4a26397a70 (127.0.98.1:38565)
I20260927 12:40:45.830397 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:45.825539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58101 (local address 127.0.98.1:38565)
0927 12:40:45.825722 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:45.825728 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:45.825749 (+ 21us) server_negotiation.cc:407] Connection header received
0927 12:40:45.825906 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:45.825912 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:45.825978 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:45.826074 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:40:45.826416 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:45.827341 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:45.829006 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:45.829731 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:45.829850 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:45.829919 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:45.830189 (+ 270us) server_negotiation.cc:299] Negotiation successful
0927 12:40:45.830259 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260927 12:40:45.830391 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:45.825695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41657 (local address 127.0.98.2:42433)
0927 12:40:45.825869 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:45.825873 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:45.826257 (+ 384us) server_negotiation.cc:407] Connection header received
0927 12:40:45.826341 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:45.826343 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:45.826422 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:45.826533 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:40:45.826780 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:45.827583 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:45.829006 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:45.829730 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:45.829850 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:45.829919 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:45.830177 (+ 258us) server_negotiation.cc:299] Negotiation successful
0927 12:40:45.830256 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 12:40:45.830397 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:45.825436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38565 (local address 127.0.98.3:58101)
0927 12:40:45.825614 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 12:40:45.825633 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 12:40:45.825756 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:40:45.826130 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:40:45.826134 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:40:45.826153 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:40:45.826295 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:45.826303 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:45.827484 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:45.827506 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:45.828854 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:45.828864 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:45.829877 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:45.829881 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:45.830059 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:45.830086 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 12:40:45.830155 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 12:40:45.830213 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":6}
I20260927 12:40:45.830404 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:45.825436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42433 (local address 127.0.98.3:41657)
0927 12:40:45.826157 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0927 12:40:45.826170 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:40:45.826263 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:40:45.826543 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:40:45.826545 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:40:45.826565 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:40:45.826651 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:45.826655 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:45.827697 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:45.827700 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:45.828870 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:45.828875 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:45.829877 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:45.829881 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:45.830017 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:45.830045 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 12:40:45.830130 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 12:40:45.830213 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"mutex_wait_us":375,"thread_start_us":168,"threads_started":1}
I20260927 12:40:45.831173 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c53ebf19f5c740a49fce5f1b88ba8b7c" candidate_uuid: "7da678c8cb71478298e71b56458c84a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "024acc06f080426da5f1aa4a26397a70" is_pre_election: true
I20260927 12:40:45.831182 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c53ebf19f5c740a49fce5f1b88ba8b7c" candidate_uuid: "7da678c8cb71478298e71b56458c84a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81b0eab12dac4fc995d9679a4c3515bf" is_pre_election: true
I20260927 12:40:45.831487 1241 raft_consensus.cc:2454] T c53ebf19f5c740a49fce5f1b88ba8b7c P 81b0eab12dac4fc995d9679a4c3515bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7da678c8cb71478298e71b56458c84a8 in term 0.
I20260927 12:40:45.831492 1107 raft_consensus.cc:2454] T c53ebf19f5c740a49fce5f1b88ba8b7c P 024acc06f080426da5f1aa4a26397a70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7da678c8cb71478298e71b56458c84a8 in term 0.
I20260927 12:40:45.831965 1310 leader_election.cc:302] T c53ebf19f5c740a49fce5f1b88ba8b7c P 7da678c8cb71478298e71b56458c84a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7da678c8cb71478298e71b56458c84a8, 81b0eab12dac4fc995d9679a4c3515bf; no voters:
I20260927 12:40:45.832240 1445 raft_consensus.cc:2781] T c53ebf19f5c740a49fce5f1b88ba8b7c P 7da678c8cb71478298e71b56458c84a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:40:45.832324 1445 raft_consensus.cc:482] T c53ebf19f5c740a49fce5f1b88ba8b7c P 7da678c8cb71478298e71b56458c84a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:40:45.832365 1445 raft_consensus.cc:3037] T c53ebf19f5c740a49fce5f1b88ba8b7c P 7da678c8cb71478298e71b56458c84a8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:40:45.833436 1445 raft_consensus.cc:504] T c53ebf19f5c740a49fce5f1b88ba8b7c P 7da678c8cb71478298e71b56458c84a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81b0eab12dac4fc995d9679a4c3515bf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42433 } } peers { permanent_uuid: "024acc06f080426da5f1aa4a26397a70" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38565 } } peers { permanent_uuid: "7da678c8cb71478298e71b56458c84a8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43215 } }
I20260927 12:40:45.833890 1445 leader_election.cc:288] T c53ebf19f5c740a49fce5f1b88ba8b7c P 7da678c8cb71478298e71b56458c84a8 [CANDIDATE]: Term 1 election: Requested vote from peers 81b0eab12dac4fc995d9679a4c3515bf (127.0.98.2:42433), 024acc06f080426da5f1aa4a26397a70 (127.0.98.1:38565)
I20260927 12:40:45.834282 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c53ebf19f5c740a49fce5f1b88ba8b7c" candidate_uuid: "7da678c8cb71478298e71b56458c84a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81b0eab12dac4fc995d9679a4c3515bf"
I20260927 12:40:45.834282 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c53ebf19f5c740a49fce5f1b88ba8b7c" candidate_uuid: "7da678c8cb71478298e71b56458c84a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "024acc06f080426da5f1aa4a26397a70"
I20260927 12:40:45.834398 1241 raft_consensus.cc:3037] T c53ebf19f5c740a49fce5f1b88ba8b7c P 81b0eab12dac4fc995d9679a4c3515bf [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:40:45.834407 1107 raft_consensus.cc:3037] T c53ebf19f5c740a49fce5f1b88ba8b7c P 024acc06f080426da5f1aa4a26397a70 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:40:45.835690 1241 raft_consensus.cc:2454] T c53ebf19f5c740a49fce5f1b88ba8b7c P 81b0eab12dac4fc995d9679a4c3515bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7da678c8cb71478298e71b56458c84a8 in term 1.
I20260927 12:40:45.835690 1107 raft_consensus.cc:2454] T c53ebf19f5c740a49fce5f1b88ba8b7c P 024acc06f080426da5f1aa4a26397a70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7da678c8cb71478298e71b56458c84a8 in term 1.
I20260927 12:40:45.836009 1310 leader_election.cc:302] T c53ebf19f5c740a49fce5f1b88ba8b7c P 7da678c8cb71478298e71b56458c84a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7da678c8cb71478298e71b56458c84a8, 81b0eab12dac4fc995d9679a4c3515bf; no voters:
I20260927 12:40:45.836308 1445 raft_consensus.cc:2781] T c53ebf19f5c740a49fce5f1b88ba8b7c P 7da678c8cb71478298e71b56458c84a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:40:45.836575 1445 raft_consensus.cc:686] T c53ebf19f5c740a49fce5f1b88ba8b7c P 7da678c8cb71478298e71b56458c84a8 [term 1 LEADER]: Becoming Leader. State: Replica: 7da678c8cb71478298e71b56458c84a8, State: Running, Role: LEADER
I20260927 12:40:45.836925 1445 consensus_queue.cc:237] T c53ebf19f5c740a49fce5f1b88ba8b7c P 7da678c8cb71478298e71b56458c84a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81b0eab12dac4fc995d9679a4c3515bf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42433 } } peers { permanent_uuid: "024acc06f080426da5f1aa4a26397a70" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38565 } } peers { permanent_uuid: "7da678c8cb71478298e71b56458c84a8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43215 } }
I20260927 12:40:45.840469 962 catalog_manager.cc:5495] T c53ebf19f5c740a49fce5f1b88ba8b7c P 7da678c8cb71478298e71b56458c84a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7da678c8cb71478298e71b56458c84a8 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "7da678c8cb71478298e71b56458c84a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81b0eab12dac4fc995d9679a4c3515bf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "024acc06f080426da5f1aa4a26397a70" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7da678c8cb71478298e71b56458c84a8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43215 } health_report { overall_health: HEALTHY } } }
Sep 27 12:40:45 dist-test-slave-bnc1 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512844, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 12:40:45.883509 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:45.874393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33100 (local address 127.0.98.1:38565)
0927 12:40:45.874558 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:45.874562 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:45.874609 (+ 47us) server_negotiation.cc:407] Connection header received
0927 12:40:45.874725 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:45.874727 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:45.874771 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:45.874843 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:40:45.875507 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:45.875989 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:45.876849 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:45.877051 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:45.877132 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:45.879568 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:40:45.879580 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:40:45.879595 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:40:45.879618 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:40:45.881788 (+ 2170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:45.882240 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:45.882246 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:45.882247 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:45.882322 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:45.882553 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:45.882555 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:45.882556 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:45.882790 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:40:45.882943 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:45.883194 (+ 251us) server_negotiation.cc:299] Negotiation successful
0927 12:40:45.883346 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260927 12:40:45.884933 1422 tablet.cc:2369] T c53ebf19f5c740a49fce5f1b88ba8b7c P 7da678c8cb71478298e71b56458c84a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 12:40:45 dist-test-slave-bnc1 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512845, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260927 12:40:45.908759 1154 tablet.cc:2369] T c53ebf19f5c740a49fce5f1b88ba8b7c P 024acc06f080426da5f1aa4a26397a70: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 12:40:45 dist-test-slave-bnc1 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512845, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 12:40:45.915756 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:45.906235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33114 (local address 127.0.98.1:38565)
0927 12:40:45.906409 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:45.906415 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:45.906713 (+ 298us) server_negotiation.cc:407] Connection header received
0927 12:40:45.906892 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:45.906895 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:45.906948 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:45.907044 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:40:45.907723 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:45.908243 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:45.909242 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:45.909691 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:45.909833 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:45.912546 (+ 2713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:40:45.912552 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:40:45.912559 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:40:45.912580 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:40:45.914087 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:45.914539 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:45.914545 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:45.914546 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:45.914624 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:45.914896 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:45.914900 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:45.914902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:45.915067 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:40:45.915203 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:45.915480 (+ 277us) server_negotiation.cc:299] Negotiation successful
0927 12:40:45.915598 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260927 12:40:45.917064 1288 tablet.cc:2369] T c53ebf19f5c740a49fce5f1b88ba8b7c P 81b0eab12dac4fc995d9679a4c3515bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 12:40:45 dist-test-slave-bnc1 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 12:40:45 dist-test-slave-bnc1 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 12:40:45.941169 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:45.932438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33130 (local address 127.0.98.1:38565)
0927 12:40:45.932611 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:45.932616 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:45.932797 (+ 181us) server_negotiation.cc:407] Connection header received
0927 12:40:45.932958 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:45.932962 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:45.933018 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:45.933110 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:40:45.933904 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:45.934440 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:45.935287 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:45.935507 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:45.935614 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:45.938020 (+ 2406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:40:45.938023 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:40:45.938025 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:40:45.938047 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:40:45.939502 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:45.939934 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:45.939939 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:45.939942 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:45.940021 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:45.940331 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:45.940334 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:45.940336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:45.940504 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:40:45.940645 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:45.940872 (+ 227us) server_negotiation.cc:299] Negotiation successful
0927 12:40:45.940988 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260927 12:40:45.942586 1067 tablet_service.cc:2908] Rejecting scan request for tablet c53ebf19f5c740a49fce5f1b88ba8b7c: Uninitialized: clean time has not yet been initialized
I20260927 12:40:45.945022 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 1024
I20260927 12:40:45.951090 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 1158
I20260927 12:40:45.956454 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 1292
I20260927 12:40:45.961710 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 930
2026-09-27T12:40:45Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2405 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:40:46 dist-test-slave-bnc1 krb5kdc[1462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:40:46 dist-test-slave-bnc1 krb5kdc[1462](info): set up 2 sockets
Sep 27 12:40:46 dist-test-slave-bnc1 krb5kdc[1462](info): commencing operation
krb5kdc: starting...
W20260927 12:40:47.025321 392 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/krb5kdc/test-user.keytab.
Sep 27 12:40:47 dist-test-slave-bnc1 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512847, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:40:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:40:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:40:47.146240 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46813
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46813
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:40:47.254437 1475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:40:47.254694 1475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:40:47.254735 1475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:40:47.258703 1475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:40:47.258769 1475 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 12:40:47.258788 1475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:40:47.258807 1475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:40:47.258823 1475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:40:47.258838 1475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:40:47.264057 1475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46813
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:40:47.265054 1475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:40:47.266088 1475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:40:47.274210 1480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:40:47.274223 1483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:40:47.274319 1481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:40:47.274428 1475 server_base.cc:1034] running on GCE node
I20260927 12:40:47.275019 1475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:40:47.276266 1475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:40:47.277463 1475 hybrid_clock.cc:648] HybridClock initialized: now 1790512847277437 us; error 44 us; skew 500 ppm
Sep 27 12:40:47 dist-test-slave-bnc1 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512847, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:40:47.280413 1475 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:40:47.281428 1475 webserver.cc:460] Webserver started at http://127.0.98.62:40553/ using document root <none> and password file <none>
I20260927 12:40:47.282073 1475 fs_manager.cc:359] Metadata directory not provided
I20260927 12:40:47.282173 1475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:40:47.282413 1475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:40:47.284551 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/minicluster-data/master-0/data/instance:
uuid: "299b3d658fec4c4284185eb876ec164a"
format_stamp: "Formatted at 2026-09-27 12:40:47 on dist-test-slave-bnc1"
server_key: "735443a592772e417f70d8355c8a1d85"
server_key_iv: "041910edbb0a71425d730edd121696f2"
server_key_version: "encryptionkey@0"
I20260927 12:40:47.285137 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/minicluster-data/master-0/wal/instance:
uuid: "299b3d658fec4c4284185eb876ec164a"
format_stamp: "Formatted at 2026-09-27 12:40:47 on dist-test-slave-bnc1"
server_key: "735443a592772e417f70d8355c8a1d85"
server_key_iv: "041910edbb0a71425d730edd121696f2"
server_key_version: "encryptionkey@0"
I20260927 12:40:47.288928 1475 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 12:40:47.291473 1490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:40:47.292762 1475 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 12:40:47.292912 1475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/minicluster-data/master-0/wal
uuid: "299b3d658fec4c4284185eb876ec164a"
format_stamp: "Formatted at 2026-09-27 12:40:47 on dist-test-slave-bnc1"
server_key: "735443a592772e417f70d8355c8a1d85"
server_key_iv: "041910edbb0a71425d730edd121696f2"
server_key_version: "encryptionkey@0"
I20260927 12:40:47.293012 1475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:40:47.308063 1475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:40:47.311021 1475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:40:47.311211 1475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:40:47.318562 1475 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46813
I20260927 12:40:47.318653 1552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46813 every 8 connection(s)
I20260927 12:40:47.319501 1475 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:40:47.322081 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 1475
I20260927 12:40:47.322196 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/minicluster-data/master-0/wal/instance
I20260927 12:40:47.322394 1553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:40:47.322448 392 external_mini_cluster.cc:1428] Setting key 597e698fb85d046b555af21f76a037af
I20260927 12:40:47.330067 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 299b3d658fec4c4284185eb876ec164a: Bootstrap starting.
Sep 27 12:40:47 dist-test-slave-bnc1 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512847, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:40:47.332676 1553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 299b3d658fec4c4284185eb876ec164a: Neither blocks nor log segments found. Creating new log.
I20260927 12:40:47.334123 1553 log.cc:824] T 00000000000000000000000000000000 P 299b3d658fec4c4284185eb876ec164a: Log is configured to *not* fsync() on all Append() calls
I20260927 12:40:47.336508 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 299b3d658fec4c4284185eb876ec164a: No bootstrap required, opened a new log
I20260927 12:40:47.337885 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:47.323855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54952 (local address 127.0.98.62:46813)
0927 12:40:47.324258 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:47.324276 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:47.324322 (+ 46us) server_negotiation.cc:407] Connection header received
0927 12:40:47.325263 (+ 941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:47.325288 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:47.325758 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:47.326175 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:40:47.326850 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:47.328329 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:47.329188 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:47.329933 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:47.330106 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:47.332615 (+ 2509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:40:47.332625 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:40:47.332647 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:40:47.332680 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:40:47.334746 (+ 2066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:47.335208 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:47.335214 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:47.335220 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:47.335352 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:47.335615 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:47.335618 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:47.335620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:47.336143 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:40:47.336321 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:47.336571 (+ 250us) server_negotiation.cc:299] Negotiation successful
0927 12:40:47.336913 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":111,"threads_started":1}
I20260927 12:40:47.340378 1553 raft_consensus.cc:348] T 00000000000000000000000000000000 P 299b3d658fec4c4284185eb876ec164a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "299b3d658fec4c4284185eb876ec164a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46813 } }
I20260927 12:40:47.340632 1553 raft_consensus.cc:374] T 00000000000000000000000000000000 P 299b3d658fec4c4284185eb876ec164a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:40:47.340673 1553 raft_consensus.cc:729] T 00000000000000000000000000000000 P 299b3d658fec4c4284185eb876ec164a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 299b3d658fec4c4284185eb876ec164a, State: Initialized, Role: FOLLOWER
I20260927 12:40:47.341090 1553 consensus_queue.cc:260] T 00000000000000000000000000000000 P 299b3d658fec4c4284185eb876ec164a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "299b3d658fec4c4284185eb876ec164a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46813 } }
I20260927 12:40:47.341225 1553 raft_consensus.cc:388] T 00000000000000000000000000000000 P 299b3d658fec4c4284185eb876ec164a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:40:47.341274 1553 raft_consensus.cc:482] T 00000000000000000000000000000000 P 299b3d658fec4c4284185eb876ec164a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:40:47.341346 1553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 299b3d658fec4c4284185eb876ec164a [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:40:47.342291 1553 raft_consensus.cc:504] T 00000000000000000000000000000000 P 299b3d658fec4c4284185eb876ec164a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "299b3d658fec4c4284185eb876ec164a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46813 } }
I20260927 12:40:47.342638 1553 leader_election.cc:302] T 00000000000000000000000000000000 P 299b3d658fec4c4284185eb876ec164a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 299b3d658fec4c4284185eb876ec164a; no voters:
I20260927 12:40:47.342962 1553 leader_election.cc:288] T 00000000000000000000000000000000 P 299b3d658fec4c4284185eb876ec164a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:40:47.343072 1558 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 299b3d658fec4c4284185eb876ec164a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:40:47.343338 1558 raft_consensus.cc:686] T 00000000000000000000000000000000 P 299b3d658fec4c4284185eb876ec164a [term 1 LEADER]: Becoming Leader. State: Replica: 299b3d658fec4c4284185eb876ec164a, State: Running, Role: LEADER
I20260927 12:40:47.343705 1558 consensus_queue.cc:237] T 00000000000000000000000000000000 P 299b3d658fec4c4284185eb876ec164a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "299b3d658fec4c4284185eb876ec164a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46813 } }
I20260927 12:40:47.344404 1553 sys_catalog.cc:564] T 00000000000000000000000000000000 P 299b3d658fec4c4284185eb876ec164a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:40:47.345670 1559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 299b3d658fec4c4284185eb876ec164a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "299b3d658fec4c4284185eb876ec164a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "299b3d658fec4c4284185eb876ec164a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46813 } } }
I20260927 12:40:47.345810 1559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 299b3d658fec4c4284185eb876ec164a [sys.catalog]: This master's current role is: LEADER
I20260927 12:40:47.346112 1560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 299b3d658fec4c4284185eb876ec164a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 299b3d658fec4c4284185eb876ec164a. Latest consensus state: current_term: 1 leader_uuid: "299b3d658fec4c4284185eb876ec164a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "299b3d658fec4c4284185eb876ec164a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46813 } } }
I20260927 12:40:47.346256 1560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 299b3d658fec4c4284185eb876ec164a [sys.catalog]: This master's current role is: LEADER
I20260927 12:40:47.346189 1567 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:40:47.349963 1567 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:40:47.355073 1567 catalog_manager.cc:1347] Generated new cluster ID: 6335bb4cd49646e98fe1d6fdff7021a6
I20260927 12:40:47.355144 1567 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:40:47.374464 1567 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:40:47.375453 1567 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:40:47.381433 1567 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 299b3d658fec4c4284185eb876ec164a: Generated new TSK 0
I20260927 12:40:47.382200 1567 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 12:40:47.418602 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46813
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:40:47.563802 1579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:40:47.564057 1579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:40:47.564100 1579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:40:47.568091 1579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:40:47.568152 1579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:40:47.568230 1579 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 12:40:47.568302 1579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:40:47.573455 1579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:40:47.574535 1579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:40:47.575606 1579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:40:47.585362 1584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:40:47.585362 1587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:40:47.585553 1585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:40:47.585753 1579 server_base.cc:1034] running on GCE node
I20260927 12:40:47.586237 1579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:40:47.586891 1579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:40:47.588086 1579 hybrid_clock.cc:648] HybridClock initialized: now 1790512847588033 us; error 72 us; skew 500 ppm
Sep 27 12:40:47 dist-test-slave-bnc1 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512847, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:40:47.591125 1579 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 12:40:47.592074 1579 webserver.cc:460] Webserver started at http://127.0.98.1:42853/ using document root <none> and password file <none>
I20260927 12:40:47.592648 1579 fs_manager.cc:359] Metadata directory not provided
I20260927 12:40:47.592736 1579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:40:47.592965 1579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:40:47.595067 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/minicluster-data/ts-0/data/instance:
uuid: "4b7e8cc171d64b2aa26381c2e2647392"
format_stamp: "Formatted at 2026-09-27 12:40:47 on dist-test-slave-bnc1"
server_key: "2c8c71dbec96360714423377171e3f64"
server_key_iv: "134900f46c209a29a9fe7302291b6b3d"
server_key_version: "encryptionkey@0"
I20260927 12:40:47.595628 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4b7e8cc171d64b2aa26381c2e2647392"
format_stamp: "Formatted at 2026-09-27 12:40:47 on dist-test-slave-bnc1"
server_key: "2c8c71dbec96360714423377171e3f64"
server_key_iv: "134900f46c209a29a9fe7302291b6b3d"
server_key_version: "encryptionkey@0"
I20260927 12:40:47.599643 1579 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 12:40:47.602093 1594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:40:47.603273 1579 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 12:40:47.603402 1579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/minicluster-data/ts-0/wal
uuid: "4b7e8cc171d64b2aa26381c2e2647392"
format_stamp: "Formatted at 2026-09-27 12:40:47 on dist-test-slave-bnc1"
server_key: "2c8c71dbec96360714423377171e3f64"
server_key_iv: "134900f46c209a29a9fe7302291b6b3d"
server_key_version: "encryptionkey@0"
I20260927 12:40:47.603521 1579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:40:47.632387 1579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:40:47.635857 1579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:40:47.636065 1579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:40:47.637441 1579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:40:47.637522 1579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:40:47.637606 1579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:40:47.637660 1579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 12:40:47 dist-test-slave-bnc1 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512847, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:40:47.648870 1579 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43393
I20260927 12:40:47.648900 1709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43393 every 8 connection(s)
I20260927 12:40:47.649982 1579 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:40:47.651589 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:47.638408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34381 (local address 127.0.98.62:46813)
0927 12:40:47.638586 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:47.638590 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:47.639464 (+ 874us) server_negotiation.cc:407] Connection header received
0927 12:40:47.640632 (+ 1168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:47.640636 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:47.640691 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:47.640798 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:40:47.641992 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:47.642555 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:47.643787 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:47.644118 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:47.644228 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:47.647617 (+ 3389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:40:47.647621 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:40:47.647623 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:40:47.647657 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:40:47.649278 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:47.649844 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:47.649848 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:47.649850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:47.649954 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:47.650338 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:47.650341 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:47.650343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:47.650575 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:40:47.650690 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:47.651277 (+ 587us) server_negotiation.cc:299] Negotiation successful
0927 12:40:47.651422 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 12:40:47.652303 1603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:47.638853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46813 (local address 127.0.98.1:34381)
0927 12:40:47.639299 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0927 12:40:47.639347 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 12:40:47.640391 (+ 1044us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:40:47.640982 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:40:47.640989 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:40:47.641305 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:40:47.641844 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:47.641857 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:47.642685 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:47.642691 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:47.643661 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:47.643670 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:47.644284 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:47.644289 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:47.644679 (+ 390us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:47.645379 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:40:47.645400 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:40:47.647441 (+ 2041us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:40:47.649393 (+ 1952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:40:47.649397 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:40:47.649408 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:40:47.649735 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:40:47.650080 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:40:47.650085 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:40:47.650087 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:40:47.650224 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:40:47.650699 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:40:47.650708 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:40:47.651009 (+ 301us) client_negotiation.cc:769] Sending connection context
0927 12:40:47.651221 (+ 212us) client_negotiation.cc:240] Negotiation successful
0927 12:40:47.651539 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":135,"threads_started":1}
I20260927 12:40:47.653601 1710 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46813
I20260927 12:40:47.653817 1710 heartbeater.cc:464] Registering TS with master...
I20260927 12:40:47.654299 1710 heartbeater.cc:511] Master 127.0.98.62:46813 requested a full tablet report, sending...
I20260927 12:40:47.655758 1507 ts_manager.cc:194] Registered new tserver with Master: 4b7e8cc171d64b2aa26381c2e2647392 (127.0.98.1:43393)
I20260927 12:40:47.655833 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 1579
I20260927 12:40:47.655954 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/minicluster-data/ts-0/wal/instance
I20260927 12:40:47.656250 392 external_mini_cluster.cc:1428] Setting key 06a65bf1c6bc1c2d3e68195d3d34154e
I20260927 12:40:47.657213 1507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34381
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 12:40:47.665524 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:47.656027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41033 (local address 127.0.98.62:46813)
0927 12:40:47.656140 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:47.656144 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:47.658066 (+ 1922us) server_negotiation.cc:407] Connection header received
0927 12:40:47.658212 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:47.658220 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:47.658259 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:47.658347 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:40:47.658810 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:47.659375 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:47.660652 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:47.660884 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:47.660994 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:47.662366 (+ 1372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:40:47.662369 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:40:47.662372 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:40:47.662396 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:40:47.663859 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:47.664307 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:47.664311 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:47.664313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:47.664417 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:47.664698 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:47.664704 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:47.664705 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:47.664879 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:40:47.664986 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:47.665250 (+ 264us) server_negotiation.cc:299] Negotiation successful
0927 12:40:47.665360 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 12:40:47.665536 1716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:47.655957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46813 (local address 127.0.98.1:41033)
0927 12:40:47.657969 (+ 2012us) negotiation.cc:107] Waiting for socket to connect
0927 12:40:47.657984 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:40:47.658086 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:40:47.658361 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:40:47.658364 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:40:47.658569 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:40:47.658687 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:47.658693 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:47.659539 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:47.659545 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:47.660514 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:47.660528 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:47.660998 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:47.661002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:47.661099 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:47.661751 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:40:47.661774 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:40:47.662152 (+ 378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:40:47.663999 (+ 1847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:40:47.664003 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:40:47.664005 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:40:47.664207 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:40:47.664514 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:40:47.664519 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:40:47.664521 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:40:47.664608 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:40:47.664986 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:40:47.664989 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:40:47.665085 (+ 96us) client_negotiation.cc:769] Sending connection context
0927 12:40:47.665188 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 12:40:47.665332 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1926,"mutex_wait_us":7,"thread_start_us":92,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 12:40:47.684594 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46813
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:40:47.794322 1719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:40:47.794667 1719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:40:47.794716 1719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:40:47.798875 1719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:40:47.798938 1719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:40:47.798978 1719 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 12:40:47.799010 1719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:40:47.804244 1719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:40:47.805195 1719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:40:47.806253 1719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:40:47.815644 1724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:40:47.815676 1727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:40:47.815682 1725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:40:47.815969 1719 server_base.cc:1034] running on GCE node
I20260927 12:40:47.816286 1719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:40:47.816856 1719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:40:47.818032 1719 hybrid_clock.cc:648] HybridClock initialized: now 1790512847818018 us; error 28 us; skew 500 ppm
Sep 27 12:40:47 dist-test-slave-bnc1 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512847, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:40:47.820806 1719 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 12:40:47.821772 1719 webserver.cc:460] Webserver started at http://127.0.98.2:36569/ using document root <none> and password file <none>
I20260927 12:40:47.822275 1719 fs_manager.cc:359] Metadata directory not provided
I20260927 12:40:47.822330 1719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:40:47.822489 1719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:40:47.824386 1719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/minicluster-data/ts-1/data/instance:
uuid: "8ac23381f96d4614af99bc2e12a4e8d3"
format_stamp: "Formatted at 2026-09-27 12:40:47 on dist-test-slave-bnc1"
server_key: "57c2298616d34817bb7aceef92263686"
server_key_iv: "5a003fd5420fbac19048b7b50f893d18"
server_key_version: "encryptionkey@0"
I20260927 12:40:47.824873 1719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8ac23381f96d4614af99bc2e12a4e8d3"
format_stamp: "Formatted at 2026-09-27 12:40:47 on dist-test-slave-bnc1"
server_key: "57c2298616d34817bb7aceef92263686"
server_key_iv: "5a003fd5420fbac19048b7b50f893d18"
server_key_version: "encryptionkey@0"
I20260927 12:40:47.828685 1719 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 12:40:47.831055 1734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:40:47.832193 1719 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 12:40:47.832280 1719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/minicluster-data/ts-1/wal
uuid: "8ac23381f96d4614af99bc2e12a4e8d3"
format_stamp: "Formatted at 2026-09-27 12:40:47 on dist-test-slave-bnc1"
server_key: "57c2298616d34817bb7aceef92263686"
server_key_iv: "5a003fd5420fbac19048b7b50f893d18"
server_key_version: "encryptionkey@0"
I20260927 12:40:47.832357 1719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:40:47.855584 1719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:40:47.858906 1719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:40:47.859059 1719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:40:47.860533 1719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:40:47.860610 1719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:40:47.860675 1719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:40:47.860713 1719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 12:40:47 dist-test-slave-bnc1 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512847, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:40:47.871618 1719 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38259
I20260927 12:40:47.871651 1849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38259 every 8 connection(s)
I20260927 12:40:47.872763 1719 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:40:47.874820 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:47.861248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43381 (local address 127.0.98.62:46813)
0927 12:40:47.861375 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:47.861379 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:47.862469 (+ 1090us) server_negotiation.cc:407] Connection header received
0927 12:40:47.863576 (+ 1107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:47.863582 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:47.863648 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:47.863733 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:40:47.865626 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:47.866538 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:47.867874 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:47.868287 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:47.868416 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:47.871141 (+ 2725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:40:47.871145 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:40:47.871148 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:40:47.871174 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:40:47.872865 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:47.873388 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:47.873391 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:47.873393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:47.873481 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:47.873885 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:47.873888 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:47.873890 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:47.874081 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:40:47.874164 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:47.874600 (+ 436us) server_negotiation.cc:299] Negotiation successful
0927 12:40:47.874703 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 12:40:47.875588 1743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:47.861830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46813 (local address 127.0.98.2:43381)
0927 12:40:47.862319 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0927 12:40:47.862363 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 12:40:47.863281 (+ 918us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:40:47.864213 (+ 932us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:40:47.864222 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:40:47.864728 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:40:47.865415 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:47.865440 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:47.866683 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:47.866687 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:47.867770 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:47.867780 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:47.868458 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:47.868464 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:47.868650 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:47.869348 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:40:47.869365 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:40:47.870934 (+ 1569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:40:47.872968 (+ 2034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:40:47.872973 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:40:47.872984 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:40:47.873270 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:40:47.873600 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:40:47.873604 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:40:47.873606 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:40:47.873765 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:40:47.874177 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:40:47.874181 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:40:47.874385 (+ 204us) client_negotiation.cc:769] Sending connection context
0927 12:40:47.874523 (+ 138us) client_negotiation.cc:240] Negotiation successful
0927 12:40:47.874769 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":143,"threads_started":1}
I20260927 12:40:47.876840 1850 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46813
I20260927 12:40:47.876988 1850 heartbeater.cc:464] Registering TS with master...
I20260927 12:40:47.877482 1850 heartbeater.cc:511] Master 127.0.98.62:46813 requested a full tablet report, sending...
I20260927 12:40:47.878481 1506 ts_manager.cc:194] Registered new tserver with Master: 8ac23381f96d4614af99bc2e12a4e8d3 (127.0.98.2:38259)
I20260927 12:40:47.879163 1506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43381
I20260927 12:40:47.880966 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 1719
I20260927 12:40:47.881105 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/minicluster-data/ts-1/wal/instance
I20260927 12:40:47.881397 392 external_mini_cluster.cc:1428] Setting key 7de803ac3cf9623d9150e4c5b80c1cac
I20260927 12:40:47.886633 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:47.879026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36765 (local address 127.0.98.62:46813)
0927 12:40:47.879146 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:47.879150 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:47.879326 (+ 176us) server_negotiation.cc:407] Connection header received
0927 12:40:47.879414 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:47.879417 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:47.879462 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:47.879534 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:40:47.880042 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:47.880771 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:47.881769 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:47.882025 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:47.882153 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:47.883434 (+ 1281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:40:47.883438 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:40:47.883440 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:40:47.883464 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:40:47.884939 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:47.885364 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:47.885369 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:47.885371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:47.885454 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:47.885766 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:47.885771 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:47.885773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:47.885987 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:40:47.886094 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:47.886355 (+ 261us) server_negotiation.cc:299] Negotiation successful
0927 12:40:47.886461 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 12:40:47.886639 1856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:47.878915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46813 (local address 127.0.98.2:36765)
0927 12:40:47.879207 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0927 12:40:47.879226 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 12:40:47.879335 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:40:47.879638 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:40:47.879641 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:40:47.879835 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:40:47.879929 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:47.879935 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:47.880920 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:47.880923 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:47.881679 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:47.881688 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:47.882280 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:47.882282 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:47.882350 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:47.882930 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:40:47.882943 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:40:47.883290 (+ 347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:40:47.885060 (+ 1770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:40:47.885064 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:40:47.885067 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:40:47.885265 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:40:47.885553 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:40:47.885558 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:40:47.885560 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:40:47.885675 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:40:47.886094 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:40:47.886098 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:40:47.886195 (+ 97us) client_negotiation.cc:769] Sending connection context
0927 12:40:47.886301 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 12:40:47.886451 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":102,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 12:40:47.908288 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46813
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:40:48.016959 1859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:40:48.017225 1859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:40:48.017300 1859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:40:48.021755 1859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:40:48.021814 1859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:40:48.021883 1859 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 12:40:48.021942 1859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:40:48.027987 1859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:40:48.029109 1859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:40:48.030254 1859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:40:48.040104 1864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:40:48.040186 1865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:40:48.040119 1867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:40:48.040346 1859 server_base.cc:1034] running on GCE node
I20260927 12:40:48.040743 1859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:40:48.041307 1859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:40:48.042460 1859 hybrid_clock.cc:648] HybridClock initialized: now 1790512848042436 us; error 45 us; skew 500 ppm
Sep 27 12:40:48 dist-test-slave-bnc1 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512848, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:40:48.045609 1859 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 12:40:48.046555 1859 webserver.cc:460] Webserver started at http://127.0.98.3:40173/ using document root <none> and password file <none>
I20260927 12:40:48.047112 1859 fs_manager.cc:359] Metadata directory not provided
I20260927 12:40:48.047183 1859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:40:48.047394 1859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:40:48.049408 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/minicluster-data/ts-2/data/instance:
uuid: "d0dd96f8ac1743fab87cada20b191272"
format_stamp: "Formatted at 2026-09-27 12:40:48 on dist-test-slave-bnc1"
server_key: "cb5b1c3142336523111aa9043e5e2b43"
server_key_iv: "d2d5bd54fb6c52f89df1cbfdab270e8a"
server_key_version: "encryptionkey@0"
I20260927 12:40:48.050030 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d0dd96f8ac1743fab87cada20b191272"
format_stamp: "Formatted at 2026-09-27 12:40:48 on dist-test-slave-bnc1"
server_key: "cb5b1c3142336523111aa9043e5e2b43"
server_key_iv: "d2d5bd54fb6c52f89df1cbfdab270e8a"
server_key_version: "encryptionkey@0"
I20260927 12:40:48.054072 1859 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 12:40:48.056551 1874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:40:48.057833 1859 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 12:40:48.057922 1859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/minicluster-data/ts-2/wal
uuid: "d0dd96f8ac1743fab87cada20b191272"
format_stamp: "Formatted at 2026-09-27 12:40:48 on dist-test-slave-bnc1"
server_key: "cb5b1c3142336523111aa9043e5e2b43"
server_key_iv: "d2d5bd54fb6c52f89df1cbfdab270e8a"
server_key_version: "encryptionkey@0"
I20260927 12:40:48.058003 1859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:40:48.077728 1859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:40:48.080705 1859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:40:48.080854 1859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:40:48.082348 1859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:40:48.082401 1859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:40:48.082489 1859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:40:48.082518 1859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 12:40:48 dist-test-slave-bnc1 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512848, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:40:48.093284 1859 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42117
I20260927 12:40:48.093355 1989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42117 every 8 connection(s)
I20260927 12:40:48.094319 1859 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:40:48.097388 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:48.083275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45147 (local address 127.0.98.62:46813)
0927 12:40:48.083451 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:48.083456 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:48.084440 (+ 984us) server_negotiation.cc:407] Connection header received
0927 12:40:48.085846 (+ 1406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:48.085851 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:48.085921 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:48.086061 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:40:48.087457 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:48.088139 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:48.089537 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:48.089858 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:48.090026 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:48.093188 (+ 3162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:40:48.093193 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:40:48.093195 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:40:48.093266 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:40:48.095347 (+ 2081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:48.095850 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:48.095853 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:48.095855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:48.095936 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:48.096283 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:48.096287 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:48.096289 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:48.096560 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:40:48.096654 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:48.097128 (+ 474us) server_negotiation.cc:299] Negotiation successful
0927 12:40:48.097237 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 12:40:48.098196 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:48.083690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46813 (local address 127.0.98.3:45147)
0927 12:40:48.084261 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0927 12:40:48.084318 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 12:40:48.085531 (+ 1213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:40:48.086267 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:40:48.086275 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:40:48.086658 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:40:48.087280 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:48.087311 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:48.088294 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:48.088299 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:48.089372 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:48.089383 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:48.090080 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:48.090086 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:48.090280 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:48.091014 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:40:48.091033 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:40:48.092932 (+ 1899us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:40:48.095459 (+ 2527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:40:48.095464 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:40:48.095474 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:40:48.095735 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:40:48.096033 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:40:48.096035 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:40:48.096036 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:40:48.096158 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:40:48.096659 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:40:48.096664 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:40:48.096912 (+ 248us) client_negotiation.cc:769] Sending connection context
0927 12:40:48.097062 (+ 150us) client_negotiation.cc:240] Negotiation successful
0927 12:40:48.097309 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":166,"threads_started":1}
I20260927 12:40:48.099354 1990 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46813
I20260927 12:40:48.099565 1990 heartbeater.cc:464] Registering TS with master...
I20260927 12:40:48.100047 1990 heartbeater.cc:511] Master 127.0.98.62:46813 requested a full tablet report, sending...
I20260927 12:40:48.101044 1507 ts_manager.cc:194] Registered new tserver with Master: d0dd96f8ac1743fab87cada20b191272 (127.0.98.3:42117)
I20260927 12:40:48.101833 1507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45147
I20260927 12:40:48.104210 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 1859
I20260927 12:40:48.104337 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/minicluster-data/ts-2/wal/instance
I20260927 12:40:48.104641 392 external_mini_cluster.cc:1428] Setting key e171361b68194f093b30832e14740169
I20260927 12:40:48.106724 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 12:40:48.109197 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:48.101841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37041 (local address 127.0.98.62:46813)
0927 12:40:48.101974 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:48.101978 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:48.102214 (+ 236us) server_negotiation.cc:407] Connection header received
0927 12:40:48.102293 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:48.102295 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:48.102331 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:48.102433 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:40:48.102830 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:48.103549 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:48.104406 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:48.104739 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:48.104844 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:48.106072 (+ 1228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:40:48.106076 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:40:48.106079 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:40:48.106102 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:40:48.107591 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:48.108028 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:48.108032 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:48.108034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:48.108160 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:48.108427 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:48.108432 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:48.108434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:48.108596 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:40:48.108707 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:48.108943 (+ 236us) server_negotiation.cc:299] Negotiation successful
0927 12:40:48.109042 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 12:40:48.109234 1996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:48.101776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46813 (local address 127.0.98.3:37041)
0927 12:40:48.102102 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0927 12:40:48.102118 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 12:40:48.102213 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:40:48.102464 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:40:48.102466 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:40:48.102626 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:40:48.102724 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:48.102729 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:48.103696 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:48.103699 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:48.104260 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:48.104268 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:48.104843 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:48.104846 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:48.104918 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:48.105460 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:40:48.105473 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:40:48.105939 (+ 466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:40:48.107739 (+ 1800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:40:48.107742 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:40:48.107744 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:40:48.107931 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:40:48.108256 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:40:48.108259 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:40:48.108261 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:40:48.108341 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:40:48.108690 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:40:48.108693 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:40:48.108779 (+ 86us) client_negotiation.cc:769] Sending connection context
0927 12:40:48.108893 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 12:40:48.109031 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":138,"threads_started":1}
Sep 27 12:40:48 dist-test-slave-bnc1 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512848, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 12:40:48 dist-test-slave-bnc1 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512848, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:40:48.131552 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:48.123954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54968 (local address 127.0.98.62:46813)
0927 12:40:48.124078 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:48.124082 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:48.124096 (+ 14us) server_negotiation.cc:407] Connection header received
0927 12:40:48.124133 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:48.124136 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:48.124173 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:48.124249 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:40:48.124942 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:48.125441 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:48.126338 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:48.126546 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:48.126624 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:48.128915 (+ 2291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:40:48.128918 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:40:48.128920 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:40:48.128940 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:40:48.130229 (+ 1289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:48.130601 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:48.130604 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:48.130605 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:48.130670 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:48.130901 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:48.130903 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:48.130904 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:48.131056 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:40:48.131153 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:48.131347 (+ 194us) server_negotiation.cc:299] Negotiation successful
0927 12:40:48.131434 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 12:40:48.134351 1507 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54968:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 12:40:48.137706 1507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:40:48.152364 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:48.146026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43393 (local address 127.0.0.1:41656)
0927 12:40:48.146709 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0927 12:40:48.146757 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 12:40:48.146887 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:40:48.148033 (+ 1146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:40:48.148036 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:40:48.148051 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:40:48.148151 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:48.148157 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:48.149450 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:48.149453 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:48.150724 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:48.150734 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:48.151820 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:48.151824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:48.151933 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:48.151977 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 12:40:48.152066 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 12:40:48.152200 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"mutex_wait_us":19,"thread_start_us":89,"threads_started":1}
I20260927 12:40:48.152554 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:48.146360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42117 (local address 127.0.0.1:46762)
0927 12:40:48.147272 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0927 12:40:48.147282 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0927 12:40:48.147394 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:40:48.147877 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:40:48.147883 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:40:48.147904 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:40:48.148027 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:48.148033 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:48.150378 (+ 2345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:48.150381 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:48.151428 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:48.151435 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:48.152190 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:48.152193 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:48.152280 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:48.152301 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 12:40:48.152363 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 12:40:48.152431 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":859,"thread_start_us":77,"threads_started":1}
I20260927 12:40:48.152865 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:48.146102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38259 (local address 127.0.0.1:33052)
0927 12:40:48.146880 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0927 12:40:48.146894 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:40:48.146984 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:40:48.148145 (+ 1161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:40:48.148147 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:40:48.148161 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:40:48.148253 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:48.148258 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:48.149189 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:48.149196 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:48.151160 (+ 1964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:48.151166 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:48.152441 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:48.152444 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:48.152550 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:48.152580 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 12:40:48.152653 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 12:40:48.152719 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":716,"mutex_wait_us":27,"thread_start_us":102,"threads_started":1}
I20260927 12:40:48.153142 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:48.146854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46762 (local address 127.0.98.3:42117)
0927 12:40:48.147169 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:48.147183 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:48.147375 (+ 192us) server_negotiation.cc:407] Connection header received
0927 12:40:48.147503 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:48.147508 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:48.147720 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:48.147874 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:40:48.148149 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:48.149343 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:48.151536 (+ 2193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:48.152069 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:48.152193 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:48.152368 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:48.152419 (+ 51us) server_negotiation.cc:299] Negotiation successful
0927 12:40:48.152966 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":106,"threads_started":1}
I20260927 12:40:48.153393 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:48.146840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33052 (local address 127.0.98.2:38259)
0927 12:40:48.147403 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:48.147414 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:48.147435 (+ 21us) server_negotiation.cc:407] Connection header received
0927 12:40:48.147508 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:48.147516 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:48.147721 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:48.147875 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:40:48.148361 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:48.149082 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:48.151300 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:48.152132 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:48.152229 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:48.152377 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:48.153144 (+ 767us) server_negotiation.cc:299] Negotiation successful
0927 12:40:48.153241 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":113,"threads_started":1}
I20260927 12:40:48.154021 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:48.146208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41656 (local address 127.0.98.1:43393)
0927 12:40:48.147627 (+ 1419us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:48.147645 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:48.147664 (+ 19us) server_negotiation.cc:407] Connection header received
0927 12:40:48.147730 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:48.147736 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:48.147887 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:48.148061 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:40:48.148244 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:48.149338 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:48.150847 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:48.151681 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:48.152967 (+ 1286us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:48.153747 (+ 780us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:48.153795 (+ 48us) server_negotiation.cc:299] Negotiation successful
0927 12:40:48.153876 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1315,"thread_start_us":99,"threads_started":1}
I20260927 12:40:48.155967 1784 tablet_service.cc:1467] Processing CreateTablet for tablet 1f0fff74855d4208a93d58b7d2acc6e6 (DEFAULT_TABLE table=test-table [id=19e993b844194337aa29202fe886ba85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:40:48.156096 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 1f0fff74855d4208a93d58b7d2acc6e6 (DEFAULT_TABLE table=test-table [id=19e993b844194337aa29202fe886ba85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:40:48.156888 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f0fff74855d4208a93d58b7d2acc6e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:40:48.156507 1644 tablet_service.cc:1467] Processing CreateTablet for tablet 1f0fff74855d4208a93d58b7d2acc6e6 (DEFAULT_TABLE table=test-table [id=19e993b844194337aa29202fe886ba85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:40:48.157160 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f0fff74855d4208a93d58b7d2acc6e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:40:48.157394 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f0fff74855d4208a93d58b7d2acc6e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:40:48.164031 2013 tablet_bootstrap.cc:492] T 1f0fff74855d4208a93d58b7d2acc6e6 P 8ac23381f96d4614af99bc2e12a4e8d3: Bootstrap starting.
I20260927 12:40:48.165920 2014 tablet_bootstrap.cc:492] T 1f0fff74855d4208a93d58b7d2acc6e6 P 4b7e8cc171d64b2aa26381c2e2647392: Bootstrap starting.
I20260927 12:40:48.166826 2013 tablet_bootstrap.cc:654] T 1f0fff74855d4208a93d58b7d2acc6e6 P 8ac23381f96d4614af99bc2e12a4e8d3: Neither blocks nor log segments found. Creating new log.
I20260927 12:40:48.167685 2013 log.cc:824] T 1f0fff74855d4208a93d58b7d2acc6e6 P 8ac23381f96d4614af99bc2e12a4e8d3: Log is configured to *not* fsync() on all Append() calls
I20260927 12:40:48.168011 2014 tablet_bootstrap.cc:654] T 1f0fff74855d4208a93d58b7d2acc6e6 P 4b7e8cc171d64b2aa26381c2e2647392: Neither blocks nor log segments found. Creating new log.
I20260927 12:40:48.168462 2015 tablet_bootstrap.cc:492] T 1f0fff74855d4208a93d58b7d2acc6e6 P d0dd96f8ac1743fab87cada20b191272: Bootstrap starting.
I20260927 12:40:48.169224 2014 log.cc:824] T 1f0fff74855d4208a93d58b7d2acc6e6 P 4b7e8cc171d64b2aa26381c2e2647392: Log is configured to *not* fsync() on all Append() calls
I20260927 12:40:48.170316 2013 tablet_bootstrap.cc:492] T 1f0fff74855d4208a93d58b7d2acc6e6 P 8ac23381f96d4614af99bc2e12a4e8d3: No bootstrap required, opened a new log
I20260927 12:40:48.170536 2013 ts_tablet_manager.cc:1397] T 1f0fff74855d4208a93d58b7d2acc6e6 P 8ac23381f96d4614af99bc2e12a4e8d3: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260927 12:40:48.170569 2015 tablet_bootstrap.cc:654] T 1f0fff74855d4208a93d58b7d2acc6e6 P d0dd96f8ac1743fab87cada20b191272: Neither blocks nor log segments found. Creating new log.
I20260927 12:40:48.171206 2014 tablet_bootstrap.cc:492] T 1f0fff74855d4208a93d58b7d2acc6e6 P 4b7e8cc171d64b2aa26381c2e2647392: No bootstrap required, opened a new log
I20260927 12:40:48.171339 2015 log.cc:824] T 1f0fff74855d4208a93d58b7d2acc6e6 P d0dd96f8ac1743fab87cada20b191272: Log is configured to *not* fsync() on all Append() calls
I20260927 12:40:48.171401 2014 ts_tablet_manager.cc:1397] T 1f0fff74855d4208a93d58b7d2acc6e6 P 4b7e8cc171d64b2aa26381c2e2647392: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 12:40:48.173216 2015 tablet_bootstrap.cc:492] T 1f0fff74855d4208a93d58b7d2acc6e6 P d0dd96f8ac1743fab87cada20b191272: No bootstrap required, opened a new log
I20260927 12:40:48.173408 2015 ts_tablet_manager.cc:1397] T 1f0fff74855d4208a93d58b7d2acc6e6 P d0dd96f8ac1743fab87cada20b191272: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 12:40:48.173928 2013 raft_consensus.cc:348] T 1f0fff74855d4208a93d58b7d2acc6e6 P 8ac23381f96d4614af99bc2e12a4e8d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b7e8cc171d64b2aa26381c2e2647392" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43393 } } peers { permanent_uuid: "8ac23381f96d4614af99bc2e12a4e8d3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38259 } } peers { permanent_uuid: "d0dd96f8ac1743fab87cada20b191272" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42117 } }
I20260927 12:40:48.174180 2013 raft_consensus.cc:374] T 1f0fff74855d4208a93d58b7d2acc6e6 P 8ac23381f96d4614af99bc2e12a4e8d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:40:48.174239 2013 raft_consensus.cc:729] T 1f0fff74855d4208a93d58b7d2acc6e6 P 8ac23381f96d4614af99bc2e12a4e8d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ac23381f96d4614af99bc2e12a4e8d3, State: Initialized, Role: FOLLOWER
I20260927 12:40:48.174696 2013 consensus_queue.cc:260] T 1f0fff74855d4208a93d58b7d2acc6e6 P 8ac23381f96d4614af99bc2e12a4e8d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b7e8cc171d64b2aa26381c2e2647392" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43393 } } peers { permanent_uuid: "8ac23381f96d4614af99bc2e12a4e8d3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38259 } } peers { permanent_uuid: "d0dd96f8ac1743fab87cada20b191272" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42117 } }
I20260927 12:40:48.175362 1850 heartbeater.cc:503] Master 127.0.98.62:46813 was elected leader, sending a full tablet report...
I20260927 12:40:48.175740 2013 ts_tablet_manager.cc:1428] T 1f0fff74855d4208a93d58b7d2acc6e6 P 8ac23381f96d4614af99bc2e12a4e8d3: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 12:40:48.176425 2014 raft_consensus.cc:348] T 1f0fff74855d4208a93d58b7d2acc6e6 P 4b7e8cc171d64b2aa26381c2e2647392 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b7e8cc171d64b2aa26381c2e2647392" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43393 } } peers { permanent_uuid: "8ac23381f96d4614af99bc2e12a4e8d3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38259 } } peers { permanent_uuid: "d0dd96f8ac1743fab87cada20b191272" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42117 } }
I20260927 12:40:48.176708 2014 raft_consensus.cc:374] T 1f0fff74855d4208a93d58b7d2acc6e6 P 4b7e8cc171d64b2aa26381c2e2647392 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:40:48.176759 2014 raft_consensus.cc:729] T 1f0fff74855d4208a93d58b7d2acc6e6 P 4b7e8cc171d64b2aa26381c2e2647392 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b7e8cc171d64b2aa26381c2e2647392, State: Initialized, Role: FOLLOWER
I20260927 12:40:48.177241 2014 consensus_queue.cc:260] T 1f0fff74855d4208a93d58b7d2acc6e6 P 4b7e8cc171d64b2aa26381c2e2647392 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b7e8cc171d64b2aa26381c2e2647392" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43393 } } peers { permanent_uuid: "8ac23381f96d4614af99bc2e12a4e8d3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38259 } } peers { permanent_uuid: "d0dd96f8ac1743fab87cada20b191272" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42117 } }
I20260927 12:40:48.177263 2015 raft_consensus.cc:348] T 1f0fff74855d4208a93d58b7d2acc6e6 P d0dd96f8ac1743fab87cada20b191272 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b7e8cc171d64b2aa26381c2e2647392" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43393 } } peers { permanent_uuid: "8ac23381f96d4614af99bc2e12a4e8d3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38259 } } peers { permanent_uuid: "d0dd96f8ac1743fab87cada20b191272" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42117 } }
I20260927 12:40:48.177488 2015 raft_consensus.cc:374] T 1f0fff74855d4208a93d58b7d2acc6e6 P d0dd96f8ac1743fab87cada20b191272 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:40:48.177538 2015 raft_consensus.cc:729] T 1f0fff74855d4208a93d58b7d2acc6e6 P d0dd96f8ac1743fab87cada20b191272 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dd96f8ac1743fab87cada20b191272, State: Initialized, Role: FOLLOWER
I20260927 12:40:48.177907 1710 heartbeater.cc:503] Master 127.0.98.62:46813 was elected leader, sending a full tablet report...
I20260927 12:40:48.178265 2014 ts_tablet_manager.cc:1428] T 1f0fff74855d4208a93d58b7d2acc6e6 P 4b7e8cc171d64b2aa26381c2e2647392: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260927 12:40:48.179035 2015 consensus_queue.cc:260] T 1f0fff74855d4208a93d58b7d2acc6e6 P d0dd96f8ac1743fab87cada20b191272 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b7e8cc171d64b2aa26381c2e2647392" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43393 } } peers { permanent_uuid: "8ac23381f96d4614af99bc2e12a4e8d3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38259 } } peers { permanent_uuid: "d0dd96f8ac1743fab87cada20b191272" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42117 } }
I20260927 12:40:48.179713 1990 heartbeater.cc:503] Master 127.0.98.62:46813 was elected leader, sending a full tablet report...
I20260927 12:40:48.180794 2015 ts_tablet_manager.cc:1428] T 1f0fff74855d4208a93d58b7d2acc6e6 P d0dd96f8ac1743fab87cada20b191272: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260927 12:40:48.220052 2020 raft_consensus.cc:482] T 1f0fff74855d4208a93d58b7d2acc6e6 P 4b7e8cc171d64b2aa26381c2e2647392 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:40:48.220293 2020 raft_consensus.cc:504] T 1f0fff74855d4208a93d58b7d2acc6e6 P 4b7e8cc171d64b2aa26381c2e2647392 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b7e8cc171d64b2aa26381c2e2647392" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43393 } } peers { permanent_uuid: "8ac23381f96d4614af99bc2e12a4e8d3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38259 } } peers { permanent_uuid: "d0dd96f8ac1743fab87cada20b191272" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42117 } }
I20260927 12:40:48.221477 2020 leader_election.cc:288] T 1f0fff74855d4208a93d58b7d2acc6e6 P 4b7e8cc171d64b2aa26381c2e2647392 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ac23381f96d4614af99bc2e12a4e8d3 (127.0.98.2:38259), d0dd96f8ac1743fab87cada20b191272 (127.0.98.3:42117)
I20260927 12:40:48.225987 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:48.221865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43525 (local address 127.0.98.3:42117)
0927 12:40:48.222014 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:48.222021 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:48.222285 (+ 264us) server_negotiation.cc:407] Connection header received
0927 12:40:48.222431 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:48.222437 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:48.222497 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:48.222578 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:40:48.222927 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:48.223648 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:48.224950 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:48.225359 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:48.225474 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:48.225537 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:48.225796 (+ 259us) server_negotiation.cc:299] Negotiation successful
0927 12:40:48.225866 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 12:40:48.225998 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:48.221895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34109 (local address 127.0.98.2:38259)
0927 12:40:48.222052 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:48.222058 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:48.222300 (+ 242us) server_negotiation.cc:407] Connection header received
0927 12:40:48.222511 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:48.222514 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:48.222583 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:48.222659 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:40:48.222953 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:48.223659 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:48.224969 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:48.225373 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:48.225490 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:48.225550 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:48.225804 (+ 254us) server_negotiation.cc:299] Negotiation successful
0927 12:40:48.225874 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 12:40:48.226022 2022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:48.221911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38259 (local address 127.0.98.1:34109)
0927 12:40:48.222188 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0927 12:40:48.222199 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 12:40:48.222398 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:40:48.222687 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:40:48.222690 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:40:48.222707 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:40:48.222824 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:48.222828 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:48.223790 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:48.223793 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:48.224836 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:48.224845 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:48.225494 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:48.225498 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:48.225675 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:48.225692 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 12:40:48.225778 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 12:40:48.225850 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":97,"thread_start_us":97,"threads_started":1}
I20260927 12:40:48.226020 2023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:48.221759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42117 (local address 127.0.98.1:43525)
0927 12:40:48.222167 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0927 12:40:48.222179 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 12:40:48.222313 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:40:48.222596 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:40:48.222600 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:40:48.222620 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:40:48.222815 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:48.222822 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:48.223754 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:48.223758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:48.224822 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:48.224831 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:48.225477 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:48.225481 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:48.225662 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:48.225681 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 12:40:48.225766 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 12:40:48.225839 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"mutex_wait_us":151,"thread_start_us":105,"threads_started":1}
I20260927 12:40:48.226780 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f0fff74855d4208a93d58b7d2acc6e6" candidate_uuid: "4b7e8cc171d64b2aa26381c2e2647392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ac23381f96d4614af99bc2e12a4e8d3" is_pre_election: true
I20260927 12:40:48.227133 1804 raft_consensus.cc:2454] T 1f0fff74855d4208a93d58b7d2acc6e6 P 8ac23381f96d4614af99bc2e12a4e8d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b7e8cc171d64b2aa26381c2e2647392 in term 0.
I20260927 12:40:48.227093 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f0fff74855d4208a93d58b7d2acc6e6" candidate_uuid: "4b7e8cc171d64b2aa26381c2e2647392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dd96f8ac1743fab87cada20b191272" is_pre_election: true
I20260927 12:40:48.227381 1944 raft_consensus.cc:2454] T 1f0fff74855d4208a93d58b7d2acc6e6 P d0dd96f8ac1743fab87cada20b191272 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b7e8cc171d64b2aa26381c2e2647392 in term 0.
I20260927 12:40:48.227679 1598 leader_election.cc:302] T 1f0fff74855d4208a93d58b7d2acc6e6 P 4b7e8cc171d64b2aa26381c2e2647392 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b7e8cc171d64b2aa26381c2e2647392, 8ac23381f96d4614af99bc2e12a4e8d3; no voters:
I20260927 12:40:48.227977 2020 raft_consensus.cc:2781] T 1f0fff74855d4208a93d58b7d2acc6e6 P 4b7e8cc171d64b2aa26381c2e2647392 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:40:48.228050 2020 raft_consensus.cc:482] T 1f0fff74855d4208a93d58b7d2acc6e6 P 4b7e8cc171d64b2aa26381c2e2647392 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:40:48.228081 2020 raft_consensus.cc:3037] T 1f0fff74855d4208a93d58b7d2acc6e6 P 4b7e8cc171d64b2aa26381c2e2647392 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:40:48.229120 2020 raft_consensus.cc:504] T 1f0fff74855d4208a93d58b7d2acc6e6 P 4b7e8cc171d64b2aa26381c2e2647392 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b7e8cc171d64b2aa26381c2e2647392" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43393 } } peers { permanent_uuid: "8ac23381f96d4614af99bc2e12a4e8d3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38259 } } peers { permanent_uuid: "d0dd96f8ac1743fab87cada20b191272" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42117 } }
I20260927 12:40:48.229487 2020 leader_election.cc:288] T 1f0fff74855d4208a93d58b7d2acc6e6 P 4b7e8cc171d64b2aa26381c2e2647392 [CANDIDATE]: Term 1 election: Requested vote from peers 8ac23381f96d4614af99bc2e12a4e8d3 (127.0.98.2:38259), d0dd96f8ac1743fab87cada20b191272 (127.0.98.3:42117)
I20260927 12:40:48.229831 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f0fff74855d4208a93d58b7d2acc6e6" candidate_uuid: "4b7e8cc171d64b2aa26381c2e2647392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dd96f8ac1743fab87cada20b191272"
I20260927 12:40:48.229965 1944 raft_consensus.cc:3037] T 1f0fff74855d4208a93d58b7d2acc6e6 P d0dd96f8ac1743fab87cada20b191272 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:40:48.230999 1944 raft_consensus.cc:2454] T 1f0fff74855d4208a93d58b7d2acc6e6 P d0dd96f8ac1743fab87cada20b191272 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b7e8cc171d64b2aa26381c2e2647392 in term 1.
I20260927 12:40:48.231308 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f0fff74855d4208a93d58b7d2acc6e6" candidate_uuid: "4b7e8cc171d64b2aa26381c2e2647392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ac23381f96d4614af99bc2e12a4e8d3"
I20260927 12:40:48.231359 1596 leader_election.cc:302] T 1f0fff74855d4208a93d58b7d2acc6e6 P 4b7e8cc171d64b2aa26381c2e2647392 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b7e8cc171d64b2aa26381c2e2647392, d0dd96f8ac1743fab87cada20b191272; no voters:
I20260927 12:40:48.231446 1804 raft_consensus.cc:3037] T 1f0fff74855d4208a93d58b7d2acc6e6 P 8ac23381f96d4614af99bc2e12a4e8d3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:40:48.231544 2020 raft_consensus.cc:2781] T 1f0fff74855d4208a93d58b7d2acc6e6 P 4b7e8cc171d64b2aa26381c2e2647392 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:40:48.231805 2020 raft_consensus.cc:686] T 1f0fff74855d4208a93d58b7d2acc6e6 P 4b7e8cc171d64b2aa26381c2e2647392 [term 1 LEADER]: Becoming Leader. State: Replica: 4b7e8cc171d64b2aa26381c2e2647392, State: Running, Role: LEADER
I20260927 12:40:48.232147 2020 consensus_queue.cc:237] T 1f0fff74855d4208a93d58b7d2acc6e6 P 4b7e8cc171d64b2aa26381c2e2647392 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b7e8cc171d64b2aa26381c2e2647392" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43393 } } peers { permanent_uuid: "8ac23381f96d4614af99bc2e12a4e8d3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38259 } } peers { permanent_uuid: "d0dd96f8ac1743fab87cada20b191272" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42117 } }
I20260927 12:40:48.233094 1804 raft_consensus.cc:2454] T 1f0fff74855d4208a93d58b7d2acc6e6 P 8ac23381f96d4614af99bc2e12a4e8d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b7e8cc171d64b2aa26381c2e2647392 in term 1.
I20260927 12:40:48.235683 1506 catalog_manager.cc:5495] T 1f0fff74855d4208a93d58b7d2acc6e6 P 4b7e8cc171d64b2aa26381c2e2647392 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b7e8cc171d64b2aa26381c2e2647392 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "4b7e8cc171d64b2aa26381c2e2647392" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b7e8cc171d64b2aa26381c2e2647392" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43393 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ac23381f96d4614af99bc2e12a4e8d3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0dd96f8ac1743fab87cada20b191272" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42117 } health_report { overall_health: UNKNOWN } } }
I20260927 12:40:48.256102 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:48.252403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41668 (local address 127.0.98.1:43393)
0927 12:40:48.252551 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:48.252554 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:48.252568 (+ 14us) server_negotiation.cc:407] Connection header received
0927 12:40:48.252749 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:48.252751 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:48.252807 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:48.252890 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:40:48.253116 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:48.253714 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:48.254618 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:48.254871 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:48.254998 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:48.255285 (+ 287us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:40:48.255703 (+ 418us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:40:48.255793 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:48.255867 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 12:40:48.255917 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 12:40:48.261960 1804 raft_consensus.cc:1261] T 1f0fff74855d4208a93d58b7d2acc6e6 P 8ac23381f96d4614af99bc2e12a4e8d3 [term 1 FOLLOWER]: Refusing update from remote peer 4b7e8cc171d64b2aa26381c2e2647392: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:40:48.262185 1944 raft_consensus.cc:1261] T 1f0fff74855d4208a93d58b7d2acc6e6 P d0dd96f8ac1743fab87cada20b191272 [term 1 FOLLOWER]: Refusing update from remote peer 4b7e8cc171d64b2aa26381c2e2647392: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:40:48.262718 2024 consensus_queue.cc:1035] T 1f0fff74855d4208a93d58b7d2acc6e6 P 4b7e8cc171d64b2aa26381c2e2647392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ac23381f96d4614af99bc2e12a4e8d3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:40:48.262884 2020 consensus_queue.cc:1035] T 1f0fff74855d4208a93d58b7d2acc6e6 P 4b7e8cc171d64b2aa26381c2e2647392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dd96f8ac1743fab87cada20b191272" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:40:48.270131 2027 mvcc.cc:205] Tried to move back new op lower bound from 7333940626473750528 to 7333940626361577472. Current Snapshot: MvccSnapshot[applied={T|T < 7333940626473750528}]
I20260927 12:40:48.283546 1507 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:40:48.283704 1507 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:40:48.286118 1507 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 299b3d658fec4c4284185eb876ec164a: Sending DeleteTablet for 3 replicas of tablet 1f0fff74855d4208a93d58b7d2acc6e6
I20260927 12:40:48.286898 1644 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f0fff74855d4208a93d58b7d2acc6e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:40:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41656
I20260927 12:40:48.287006 1784 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f0fff74855d4208a93d58b7d2acc6e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:40:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33052
I20260927 12:40:48.287168 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f0fff74855d4208a93d58b7d2acc6e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:40:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46762
I20260927 12:40:48.287410 2038 tablet_replica.cc:330] T 1f0fff74855d4208a93d58b7d2acc6e6 P 8ac23381f96d4614af99bc2e12a4e8d3: stopping tablet replica
I20260927 12:40:48.287724 2038 raft_consensus.cc:2229] T 1f0fff74855d4208a93d58b7d2acc6e6 P 8ac23381f96d4614af99bc2e12a4e8d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:40:48.288065 2038 raft_consensus.cc:2258] T 1f0fff74855d4208a93d58b7d2acc6e6 P 8ac23381f96d4614af99bc2e12a4e8d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:40:48.288129 2039 tablet_replica.cc:330] T 1f0fff74855d4208a93d58b7d2acc6e6 P d0dd96f8ac1743fab87cada20b191272: stopping tablet replica
I20260927 12:40:48.288405 2039 raft_consensus.cc:2229] T 1f0fff74855d4208a93d58b7d2acc6e6 P d0dd96f8ac1743fab87cada20b191272 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:40:48.288376 1507 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54968:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 12:40:48.288630 2039 raft_consensus.cc:2258] T 1f0fff74855d4208a93d58b7d2acc6e6 P d0dd96f8ac1743fab87cada20b191272 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 12:40:48.288766 1507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:40:48.288941 2037 tablet_replica.cc:330] T 1f0fff74855d4208a93d58b7d2acc6e6 P 4b7e8cc171d64b2aa26381c2e2647392: stopping tablet replica
I20260927 12:40:48.289201 2037 raft_consensus.cc:2229] T 1f0fff74855d4208a93d58b7d2acc6e6 P 4b7e8cc171d64b2aa26381c2e2647392 [term 1 LEADER]: Raft consensus shutting down.
I20260927 12:40:48.289644 2037 raft_consensus.cc:2258] T 1f0fff74855d4208a93d58b7d2acc6e6 P 4b7e8cc171d64b2aa26381c2e2647392 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:40:48.290306 2039 ts_tablet_manager.cc:1905] T 1f0fff74855d4208a93d58b7d2acc6e6 P d0dd96f8ac1743fab87cada20b191272: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:40:48.291399 2038 ts_tablet_manager.cc:1905] T 1f0fff74855d4208a93d58b7d2acc6e6 P 8ac23381f96d4614af99bc2e12a4e8d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:40:48.293118 2039 ts_tablet_manager.cc:1918] T 1f0fff74855d4208a93d58b7d2acc6e6 P d0dd96f8ac1743fab87cada20b191272: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:40:48.293224 2039 log.cc:1196] T 1f0fff74855d4208a93d58b7d2acc6e6 P d0dd96f8ac1743fab87cada20b191272: Deleting WAL directory at /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/minicluster-data/ts-2/wal/wals/1f0fff74855d4208a93d58b7d2acc6e6
I20260927 12:40:48.293977 2039 ts_tablet_manager.cc:1939] T 1f0fff74855d4208a93d58b7d2acc6e6 P d0dd96f8ac1743fab87cada20b191272: Deleting consensus metadata
I20260927 12:40:48.294839 1784 tablet_service.cc:1467] Processing CreateTablet for tablet 4c73652ab07441f6907f8267747f314c (DEFAULT_TABLE table=test-table [id=30e401904c1240179c1c3a3ec73968f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:40:48.294956 1644 tablet_service.cc:1467] Processing CreateTablet for tablet 4c73652ab07441f6907f8267747f314c (DEFAULT_TABLE table=test-table [id=30e401904c1240179c1c3a3ec73968f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:40:48.295234 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c73652ab07441f6907f8267747f314c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:40:48.295252 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 4c73652ab07441f6907f8267747f314c (DEFAULT_TABLE table=test-table [id=30e401904c1240179c1c3a3ec73968f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:40:48.295477 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c73652ab07441f6907f8267747f314c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:40:48.296242 2037 ts_tablet_manager.cc:1905] T 1f0fff74855d4208a93d58b7d2acc6e6 P 4b7e8cc171d64b2aa26381c2e2647392: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:40:48.297744 2015 tablet_bootstrap.cc:492] T 4c73652ab07441f6907f8267747f314c P d0dd96f8ac1743fab87cada20b191272: Bootstrap starting.
I20260927 12:40:48.298015 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c73652ab07441f6907f8267747f314c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:40:48.298746 2015 tablet_bootstrap.cc:654] T 4c73652ab07441f6907f8267747f314c P d0dd96f8ac1743fab87cada20b191272: Neither blocks nor log segments found. Creating new log.
I20260927 12:40:48.299414 2014 tablet_bootstrap.cc:492] T 4c73652ab07441f6907f8267747f314c P 4b7e8cc171d64b2aa26381c2e2647392: Bootstrap starting.
I20260927 12:40:48.300103 2013 tablet_bootstrap.cc:492] T 4c73652ab07441f6907f8267747f314c P 8ac23381f96d4614af99bc2e12a4e8d3: Bootstrap starting.
I20260927 12:40:48.300351 2014 tablet_bootstrap.cc:654] T 4c73652ab07441f6907f8267747f314c P 4b7e8cc171d64b2aa26381c2e2647392: Neither blocks nor log segments found. Creating new log.
I20260927 12:40:48.300442 2037 ts_tablet_manager.cc:1918] T 1f0fff74855d4208a93d58b7d2acc6e6 P 4b7e8cc171d64b2aa26381c2e2647392: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:40:48.300544 2037 log.cc:1196] T 1f0fff74855d4208a93d58b7d2acc6e6 P 4b7e8cc171d64b2aa26381c2e2647392: Deleting WAL directory at /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/minicluster-data/ts-0/wal/wals/1f0fff74855d4208a93d58b7d2acc6e6
I20260927 12:40:48.300907 2037 ts_tablet_manager.cc:1939] T 1f0fff74855d4208a93d58b7d2acc6e6 P 4b7e8cc171d64b2aa26381c2e2647392: Deleting consensus metadata
I20260927 12:40:48.301110 2038 ts_tablet_manager.cc:1918] T 1f0fff74855d4208a93d58b7d2acc6e6 P 8ac23381f96d4614af99bc2e12a4e8d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:40:48.301241 2038 log.cc:1196] T 1f0fff74855d4208a93d58b7d2acc6e6 P 8ac23381f96d4614af99bc2e12a4e8d3: Deleting WAL directory at /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/minicluster-data/ts-1/wal/wals/1f0fff74855d4208a93d58b7d2acc6e6
I20260927 12:40:48.301489 2013 tablet_bootstrap.cc:654] T 4c73652ab07441f6907f8267747f314c P 8ac23381f96d4614af99bc2e12a4e8d3: Neither blocks nor log segments found. Creating new log.
I20260927 12:40:48.301656 1492 catalog_manager.cc:4841] TS d0dd96f8ac1743fab87cada20b191272 (127.0.98.3:42117): tablet 1f0fff74855d4208a93d58b7d2acc6e6 (table test-table [id=19e993b844194337aa29202fe886ba85]) successfully deleted
I20260927 12:40:48.301879 2038 ts_tablet_manager.cc:1939] T 1f0fff74855d4208a93d58b7d2acc6e6 P 8ac23381f96d4614af99bc2e12a4e8d3: Deleting consensus metadata
I20260927 12:40:48.302208 1493 catalog_manager.cc:4841] TS 4b7e8cc171d64b2aa26381c2e2647392 (127.0.98.1:43393): tablet 1f0fff74855d4208a93d58b7d2acc6e6 (table test-table [id=19e993b844194337aa29202fe886ba85]) successfully deleted
I20260927 12:40:48.303064 1494 catalog_manager.cc:4841] TS 8ac23381f96d4614af99bc2e12a4e8d3 (127.0.98.2:38259): tablet 1f0fff74855d4208a93d58b7d2acc6e6 (table test-table [id=19e993b844194337aa29202fe886ba85]) successfully deleted
I20260927 12:40:48.303175 2015 tablet_bootstrap.cc:492] T 4c73652ab07441f6907f8267747f314c P d0dd96f8ac1743fab87cada20b191272: No bootstrap required, opened a new log
I20260927 12:40:48.303261 2015 ts_tablet_manager.cc:1397] T 4c73652ab07441f6907f8267747f314c P d0dd96f8ac1743fab87cada20b191272: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260927 12:40:48.303400 2014 tablet_bootstrap.cc:492] T 4c73652ab07441f6907f8267747f314c P 4b7e8cc171d64b2aa26381c2e2647392: No bootstrap required, opened a new log
I20260927 12:40:48.303483 2014 ts_tablet_manager.cc:1397] T 4c73652ab07441f6907f8267747f314c P 4b7e8cc171d64b2aa26381c2e2647392: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 12:40:48.303743 2015 raft_consensus.cc:348] T 4c73652ab07441f6907f8267747f314c P d0dd96f8ac1743fab87cada20b191272 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b7e8cc171d64b2aa26381c2e2647392" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43393 } } peers { permanent_uuid: "8ac23381f96d4614af99bc2e12a4e8d3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38259 } } peers { permanent_uuid: "d0dd96f8ac1743fab87cada20b191272" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42117 } }
I20260927 12:40:48.303874 2015 raft_consensus.cc:374] T 4c73652ab07441f6907f8267747f314c P d0dd96f8ac1743fab87cada20b191272 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:40:48.303920 2015 raft_consensus.cc:729] T 4c73652ab07441f6907f8267747f314c P d0dd96f8ac1743fab87cada20b191272 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dd96f8ac1743fab87cada20b191272, State: Initialized, Role: FOLLOWER
I20260927 12:40:48.303917 2014 raft_consensus.cc:348] T 4c73652ab07441f6907f8267747f314c P 4b7e8cc171d64b2aa26381c2e2647392 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b7e8cc171d64b2aa26381c2e2647392" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43393 } } peers { permanent_uuid: "8ac23381f96d4614af99bc2e12a4e8d3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38259 } } peers { permanent_uuid: "d0dd96f8ac1743fab87cada20b191272" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42117 } }
I20260927 12:40:48.304023 2014 raft_consensus.cc:374] T 4c73652ab07441f6907f8267747f314c P 4b7e8cc171d64b2aa26381c2e2647392 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:40:48.304054 2014 raft_consensus.cc:729] T 4c73652ab07441f6907f8267747f314c P 4b7e8cc171d64b2aa26381c2e2647392 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b7e8cc171d64b2aa26381c2e2647392, State: Initialized, Role: FOLLOWER
I20260927 12:40:48.304054 2015 consensus_queue.cc:260] T 4c73652ab07441f6907f8267747f314c P d0dd96f8ac1743fab87cada20b191272 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b7e8cc171d64b2aa26381c2e2647392" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43393 } } peers { permanent_uuid: "8ac23381f96d4614af99bc2e12a4e8d3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38259 } } peers { permanent_uuid: "d0dd96f8ac1743fab87cada20b191272" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42117 } }
I20260927 12:40:48.304171 2014 consensus_queue.cc:260] T 4c73652ab07441f6907f8267747f314c P 4b7e8cc171d64b2aa26381c2e2647392 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b7e8cc171d64b2aa26381c2e2647392" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43393 } } peers { permanent_uuid: "8ac23381f96d4614af99bc2e12a4e8d3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38259 } } peers { permanent_uuid: "d0dd96f8ac1743fab87cada20b191272" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42117 } }
I20260927 12:40:48.304332 2015 ts_tablet_manager.cc:1428] T 4c73652ab07441f6907f8267747f314c P d0dd96f8ac1743fab87cada20b191272: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260927 12:40:48.304417 2014 ts_tablet_manager.cc:1428] T 4c73652ab07441f6907f8267747f314c P 4b7e8cc171d64b2aa26381c2e2647392: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 12:40:48.306447 2013 tablet_bootstrap.cc:492] T 4c73652ab07441f6907f8267747f314c P 8ac23381f96d4614af99bc2e12a4e8d3: No bootstrap required, opened a new log
I20260927 12:40:48.306545 2013 ts_tablet_manager.cc:1397] T 4c73652ab07441f6907f8267747f314c P 8ac23381f96d4614af99bc2e12a4e8d3: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260927 12:40:48.307010 2013 raft_consensus.cc:348] T 4c73652ab07441f6907f8267747f314c P 8ac23381f96d4614af99bc2e12a4e8d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b7e8cc171d64b2aa26381c2e2647392" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43393 } } peers { permanent_uuid: "8ac23381f96d4614af99bc2e12a4e8d3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38259 } } peers { permanent_uuid: "d0dd96f8ac1743fab87cada20b191272" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42117 } }
I20260927 12:40:48.307132 2013 raft_consensus.cc:374] T 4c73652ab07441f6907f8267747f314c P 8ac23381f96d4614af99bc2e12a4e8d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:40:48.307161 2013 raft_consensus.cc:729] T 4c73652ab07441f6907f8267747f314c P 8ac23381f96d4614af99bc2e12a4e8d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ac23381f96d4614af99bc2e12a4e8d3, State: Initialized, Role: FOLLOWER
I20260927 12:40:48.307289 2013 consensus_queue.cc:260] T 4c73652ab07441f6907f8267747f314c P 8ac23381f96d4614af99bc2e12a4e8d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b7e8cc171d64b2aa26381c2e2647392" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43393 } } peers { permanent_uuid: "8ac23381f96d4614af99bc2e12a4e8d3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38259 } } peers { permanent_uuid: "d0dd96f8ac1743fab87cada20b191272" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42117 } }
I20260927 12:40:48.307578 2013 ts_tablet_manager.cc:1428] T 4c73652ab07441f6907f8267747f314c P 8ac23381f96d4614af99bc2e12a4e8d3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20260927 12:40:48.345398 1991 tablet.cc:2369] T 4c73652ab07441f6907f8267747f314c P d0dd96f8ac1743fab87cada20b191272: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 12:40:48.374048 1851 tablet.cc:2369] T 4c73652ab07441f6907f8267747f314c P 8ac23381f96d4614af99bc2e12a4e8d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 12:40:48.401291 1711 tablet.cc:2369] T 4c73652ab07441f6907f8267747f314c P 4b7e8cc171d64b2aa26381c2e2647392: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:40:48.426704 2021 raft_consensus.cc:482] T 4c73652ab07441f6907f8267747f314c P d0dd96f8ac1743fab87cada20b191272 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:40:48.426838 2021 raft_consensus.cc:504] T 4c73652ab07441f6907f8267747f314c P d0dd96f8ac1743fab87cada20b191272 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b7e8cc171d64b2aa26381c2e2647392" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43393 } } peers { permanent_uuid: "8ac23381f96d4614af99bc2e12a4e8d3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38259 } } peers { permanent_uuid: "d0dd96f8ac1743fab87cada20b191272" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42117 } }
I20260927 12:40:48.427951 2021 leader_election.cc:288] T 4c73652ab07441f6907f8267747f314c P d0dd96f8ac1743fab87cada20b191272 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4b7e8cc171d64b2aa26381c2e2647392 (127.0.98.1:43393), 8ac23381f96d4614af99bc2e12a4e8d3 (127.0.98.2:38259)
I20260927 12:40:48.432428 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:48.428373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57087 (local address 127.0.98.1:43393)
0927 12:40:48.428499 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:48.428503 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:48.428863 (+ 360us) server_negotiation.cc:407] Connection header received
0927 12:40:48.429046 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:48.429051 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:48.429096 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:48.429168 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:40:48.429452 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:48.430114 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:48.431180 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:48.431766 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:48.431884 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:48.431956 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:48.432214 (+ 258us) server_negotiation.cc:299] Negotiation successful
0927 12:40:48.432300 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 12:40:48.432433 2043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:48.428191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43393 (local address 127.0.98.3:57087)
0927 12:40:48.428780 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0927 12:40:48.428793 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:40:48.428931 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:40:48.429199 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:40:48.429202 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:40:48.429221 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:40:48.429333 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:48.429338 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:48.430220 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:48.430226 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:48.431052 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:48.431060 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:48.431908 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:48.431912 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:48.432013 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:48.432099 (+ 86us) client_negotiation.cc:769] Sending connection context
0927 12:40:48.432185 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 12:40:48.432257 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"mutex_wait_us":369,"thread_start_us":68,"threads_started":1}
I20260927 12:40:48.432469 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:48.428189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38259 (local address 127.0.98.3:35815)
0927 12:40:48.428402 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0927 12:40:48.428417 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:40:48.428525 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:40:48.428804 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:40:48.428807 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:40:48.428832 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:40:48.428949 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:48.428955 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:48.429941 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:48.429944 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:48.431187 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:48.431196 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:48.432025 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:48.432029 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:48.432196 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:48.432220 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 12:40:48.432290 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 12:40:48.432357 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":5}
I20260927 12:40:48.432935 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c73652ab07441f6907f8267747f314c" candidate_uuid: "d0dd96f8ac1743fab87cada20b191272" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b7e8cc171d64b2aa26381c2e2647392" is_pre_election: true
I20260927 12:40:48.433076 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:48.428299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35815 (local address 127.0.98.2:38259)
0927 12:40:48.428423 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:48.428429 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:48.428471 (+ 42us) server_negotiation.cc:407] Connection header received
0927 12:40:48.428642 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:48.428646 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:48.428695 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:48.428771 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:40:48.429072 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:48.429830 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:48.431319 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:48.431905 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:48.432002 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:48.432052 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:48.432889 (+ 837us) server_negotiation.cc:299] Negotiation successful
0927 12:40:48.432969 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 12:40:48.433175 1664 raft_consensus.cc:2454] T 4c73652ab07441f6907f8267747f314c P 4b7e8cc171d64b2aa26381c2e2647392 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0dd96f8ac1743fab87cada20b191272 in term 0.
I20260927 12:40:48.433459 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c73652ab07441f6907f8267747f314c" candidate_uuid: "d0dd96f8ac1743fab87cada20b191272" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ac23381f96d4614af99bc2e12a4e8d3" is_pre_election: true
I20260927 12:40:48.433635 1804 raft_consensus.cc:2454] T 4c73652ab07441f6907f8267747f314c P 8ac23381f96d4614af99bc2e12a4e8d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0dd96f8ac1743fab87cada20b191272 in term 0.
I20260927 12:40:48.433653 1877 leader_election.cc:302] T 4c73652ab07441f6907f8267747f314c P d0dd96f8ac1743fab87cada20b191272 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b7e8cc171d64b2aa26381c2e2647392, d0dd96f8ac1743fab87cada20b191272; no voters:
I20260927 12:40:48.433934 2021 raft_consensus.cc:2781] T 4c73652ab07441f6907f8267747f314c P d0dd96f8ac1743fab87cada20b191272 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:40:48.434012 2021 raft_consensus.cc:482] T 4c73652ab07441f6907f8267747f314c P d0dd96f8ac1743fab87cada20b191272 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:40:48.434060 2021 raft_consensus.cc:3037] T 4c73652ab07441f6907f8267747f314c P d0dd96f8ac1743fab87cada20b191272 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:40:48.435150 2021 raft_consensus.cc:504] T 4c73652ab07441f6907f8267747f314c P d0dd96f8ac1743fab87cada20b191272 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b7e8cc171d64b2aa26381c2e2647392" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43393 } } peers { permanent_uuid: "8ac23381f96d4614af99bc2e12a4e8d3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38259 } } peers { permanent_uuid: "d0dd96f8ac1743fab87cada20b191272" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42117 } }
I20260927 12:40:48.435532 2021 leader_election.cc:288] T 4c73652ab07441f6907f8267747f314c P d0dd96f8ac1743fab87cada20b191272 [CANDIDATE]: Term 1 election: Requested vote from peers 4b7e8cc171d64b2aa26381c2e2647392 (127.0.98.1:43393), 8ac23381f96d4614af99bc2e12a4e8d3 (127.0.98.2:38259)
I20260927 12:40:48.435843 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c73652ab07441f6907f8267747f314c" candidate_uuid: "d0dd96f8ac1743fab87cada20b191272" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b7e8cc171d64b2aa26381c2e2647392"
I20260927 12:40:48.435935 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c73652ab07441f6907f8267747f314c" candidate_uuid: "d0dd96f8ac1743fab87cada20b191272" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ac23381f96d4614af99bc2e12a4e8d3"
I20260927 12:40:48.435968 1664 raft_consensus.cc:3037] T 4c73652ab07441f6907f8267747f314c P 4b7e8cc171d64b2aa26381c2e2647392 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:40:48.436045 1804 raft_consensus.cc:3037] T 4c73652ab07441f6907f8267747f314c P 8ac23381f96d4614af99bc2e12a4e8d3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:40:48.436903 1664 raft_consensus.cc:2454] T 4c73652ab07441f6907f8267747f314c P 4b7e8cc171d64b2aa26381c2e2647392 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0dd96f8ac1743fab87cada20b191272 in term 1.
I20260927 12:40:48.437156 1877 leader_election.cc:302] T 4c73652ab07441f6907f8267747f314c P d0dd96f8ac1743fab87cada20b191272 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b7e8cc171d64b2aa26381c2e2647392, d0dd96f8ac1743fab87cada20b191272; no voters:
I20260927 12:40:48.437165 1804 raft_consensus.cc:2454] T 4c73652ab07441f6907f8267747f314c P 8ac23381f96d4614af99bc2e12a4e8d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0dd96f8ac1743fab87cada20b191272 in term 1.
I20260927 12:40:48.437342 2021 raft_consensus.cc:2781] T 4c73652ab07441f6907f8267747f314c P d0dd96f8ac1743fab87cada20b191272 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:40:48.437546 2021 raft_consensus.cc:686] T 4c73652ab07441f6907f8267747f314c P d0dd96f8ac1743fab87cada20b191272 [term 1 LEADER]: Becoming Leader. State: Replica: d0dd96f8ac1743fab87cada20b191272, State: Running, Role: LEADER
I20260927 12:40:48.437812 2021 consensus_queue.cc:237] T 4c73652ab07441f6907f8267747f314c P d0dd96f8ac1743fab87cada20b191272 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b7e8cc171d64b2aa26381c2e2647392" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43393 } } peers { permanent_uuid: "8ac23381f96d4614af99bc2e12a4e8d3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38259 } } peers { permanent_uuid: "d0dd96f8ac1743fab87cada20b191272" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42117 } }
I20260927 12:40:48.440114 1506 catalog_manager.cc:5495] T 4c73652ab07441f6907f8267747f314c P d0dd96f8ac1743fab87cada20b191272 reported cstate change: term changed from 0 to 1, leader changed from <none> to d0dd96f8ac1743fab87cada20b191272 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "d0dd96f8ac1743fab87cada20b191272" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b7e8cc171d64b2aa26381c2e2647392" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ac23381f96d4614af99bc2e12a4e8d3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0dd96f8ac1743fab87cada20b191272" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42117 } health_report { overall_health: HEALTHY } } }
Sep 27 12:40:48 dist-test-slave-bnc1 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512847, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:40:48.479614 2051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:48.471309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46813 (local address 127.0.0.1:54982)
0927 12:40:48.471611 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0927 12:40:48.471623 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 12:40:48.471758 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:40:48.472025 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:40:48.472027 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:40:48.472467 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:40:48.472555 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:48.472561 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:48.473286 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:48.473292 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:48.473885 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:48.473894 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:48.474311 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:48.474314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:48.474398 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:48.474999 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:40:48.475011 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:40:48.476489 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:40:48.478227 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:40:48.478236 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:40:48.478241 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:40:48.478474 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:40:48.478716 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:40:48.478718 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:40:48.478719 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:40:48.478792 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:40:48.479132 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:40:48.479137 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:40:48.479241 (+ 104us) client_negotiation.cc:769] Sending connection context
0927 12:40:48.479330 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 12:40:48.479456 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":112,"threads_started":1}
I20260927 12:40:48.479636 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:48.471336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54982 (local address 127.0.98.62:46813)
0927 12:40:48.471481 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:48.471487 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:48.471706 (+ 219us) server_negotiation.cc:407] Connection header received
0927 12:40:48.471874 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:48.471879 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:48.471920 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:48.472009 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:40:48.472673 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:48.473184 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:48.474001 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:48.474208 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:48.474311 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:48.476641 (+ 2330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:40:48.476645 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:40:48.476647 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:40:48.476669 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:40:48.478126 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:48.478562 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:48.478565 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:48.478566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:48.478639 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:48.478867 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:48.478869 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:48.478871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:48.479046 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:40:48.479122 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:48.479389 (+ 267us) server_negotiation.cc:299] Negotiation successful
0927 12:40:48.479513 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":50}
I20260927 12:40:48.483397 1506 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54982:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 12:40:48.484172 1506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:40:48.489015 1924 tablet_service.cc:1467] Processing CreateTablet for tablet af3111b92c7247058bda7dd4aa5dfca5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=463f6f7a4d6e4bd3b4a86f11610af3b1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 12:40:48.489015 1644 tablet_service.cc:1467] Processing CreateTablet for tablet af3111b92c7247058bda7dd4aa5dfca5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=463f6f7a4d6e4bd3b4a86f11610af3b1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 12:40:48.489454 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af3111b92c7247058bda7dd4aa5dfca5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:40:48.489454 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af3111b92c7247058bda7dd4aa5dfca5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:40:48.489934 1784 tablet_service.cc:1467] Processing CreateTablet for tablet af3111b92c7247058bda7dd4aa5dfca5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=463f6f7a4d6e4bd3b4a86f11610af3b1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 12:40:48.490298 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af3111b92c7247058bda7dd4aa5dfca5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:40:48.492545 2015 tablet_bootstrap.cc:492] T af3111b92c7247058bda7dd4aa5dfca5 P d0dd96f8ac1743fab87cada20b191272: Bootstrap starting.
I20260927 12:40:48.493088 2013 tablet_bootstrap.cc:492] T af3111b92c7247058bda7dd4aa5dfca5 P 8ac23381f96d4614af99bc2e12a4e8d3: Bootstrap starting.
I20260927 12:40:48.493435 2015 tablet_bootstrap.cc:654] T af3111b92c7247058bda7dd4aa5dfca5 P d0dd96f8ac1743fab87cada20b191272: Neither blocks nor log segments found. Creating new log.
I20260927 12:40:48.494011 2013 tablet_bootstrap.cc:654] T af3111b92c7247058bda7dd4aa5dfca5 P 8ac23381f96d4614af99bc2e12a4e8d3: Neither blocks nor log segments found. Creating new log.
I20260927 12:40:48.495230 2013 tablet_bootstrap.cc:492] T af3111b92c7247058bda7dd4aa5dfca5 P 8ac23381f96d4614af99bc2e12a4e8d3: No bootstrap required, opened a new log
I20260927 12:40:48.495326 2013 ts_tablet_manager.cc:1397] T af3111b92c7247058bda7dd4aa5dfca5 P 8ac23381f96d4614af99bc2e12a4e8d3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 12:40:48.495565 2015 tablet_bootstrap.cc:492] T af3111b92c7247058bda7dd4aa5dfca5 P d0dd96f8ac1743fab87cada20b191272: No bootstrap required, opened a new log
I20260927 12:40:48.495646 2015 ts_tablet_manager.cc:1397] T af3111b92c7247058bda7dd4aa5dfca5 P d0dd96f8ac1743fab87cada20b191272: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 12:40:48.495790 2013 raft_consensus.cc:348] T af3111b92c7247058bda7dd4aa5dfca5 P 8ac23381f96d4614af99bc2e12a4e8d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b7e8cc171d64b2aa26381c2e2647392" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43393 } } peers { permanent_uuid: "8ac23381f96d4614af99bc2e12a4e8d3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38259 } } peers { permanent_uuid: "d0dd96f8ac1743fab87cada20b191272" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42117 } }
I20260927 12:40:48.495918 2013 raft_consensus.cc:374] T af3111b92c7247058bda7dd4aa5dfca5 P 8ac23381f96d4614af99bc2e12a4e8d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:40:48.495959 2013 raft_consensus.cc:729] T af3111b92c7247058bda7dd4aa5dfca5 P 8ac23381f96d4614af99bc2e12a4e8d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ac23381f96d4614af99bc2e12a4e8d3, State: Initialized, Role: FOLLOWER
I20260927 12:40:48.496088 2013 consensus_queue.cc:260] T af3111b92c7247058bda7dd4aa5dfca5 P 8ac23381f96d4614af99bc2e12a4e8d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b7e8cc171d64b2aa26381c2e2647392" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43393 } } peers { permanent_uuid: "8ac23381f96d4614af99bc2e12a4e8d3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38259 } } peers { permanent_uuid: "d0dd96f8ac1743fab87cada20b191272" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42117 } }
I20260927 12:40:48.496083 2015 raft_consensus.cc:348] T af3111b92c7247058bda7dd4aa5dfca5 P d0dd96f8ac1743fab87cada20b191272 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b7e8cc171d64b2aa26381c2e2647392" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43393 } } peers { permanent_uuid: "8ac23381f96d4614af99bc2e12a4e8d3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38259 } } peers { permanent_uuid: "d0dd96f8ac1743fab87cada20b191272" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42117 } }
I20260927 12:40:48.496188 2015 raft_consensus.cc:374] T af3111b92c7247058bda7dd4aa5dfca5 P d0dd96f8ac1743fab87cada20b191272 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:40:48.496222 2015 raft_consensus.cc:729] T af3111b92c7247058bda7dd4aa5dfca5 P d0dd96f8ac1743fab87cada20b191272 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dd96f8ac1743fab87cada20b191272, State: Initialized, Role: FOLLOWER
I20260927 12:40:48.496264 2014 tablet_bootstrap.cc:492] T af3111b92c7247058bda7dd4aa5dfca5 P 4b7e8cc171d64b2aa26381c2e2647392: Bootstrap starting.
I20260927 12:40:48.496340 2015 consensus_queue.cc:260] T af3111b92c7247058bda7dd4aa5dfca5 P d0dd96f8ac1743fab87cada20b191272 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b7e8cc171d64b2aa26381c2e2647392" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43393 } } peers { permanent_uuid: "8ac23381f96d4614af99bc2e12a4e8d3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38259 } } peers { permanent_uuid: "d0dd96f8ac1743fab87cada20b191272" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42117 } }
I20260927 12:40:48.496423 2013 ts_tablet_manager.cc:1428] T af3111b92c7247058bda7dd4aa5dfca5 P 8ac23381f96d4614af99bc2e12a4e8d3: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260927 12:40:48.496457 2019 tablet_replica.cc:439] T af3111b92c7247058bda7dd4aa5dfca5 P 8ac23381f96d4614af99bc2e12a4e8d3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b7e8cc171d64b2aa26381c2e2647392" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43393 } } peers { permanent_uuid: "8ac23381f96d4614af99bc2e12a4e8d3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38259 } } peers { permanent_uuid: "d0dd96f8ac1743fab87cada20b191272" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42117 } } }
I20260927 12:40:48.496595 2019 tablet_replica.cc:442] T af3111b92c7247058bda7dd4aa5dfca5 P 8ac23381f96d4614af99bc2e12a4e8d3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 12:40:48.496622 2015 ts_tablet_manager.cc:1428] T af3111b92c7247058bda7dd4aa5dfca5 P d0dd96f8ac1743fab87cada20b191272: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 12:40:48.496858 2021 tablet_replica.cc:439] T af3111b92c7247058bda7dd4aa5dfca5 P d0dd96f8ac1743fab87cada20b191272: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b7e8cc171d64b2aa26381c2e2647392" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43393 } } peers { permanent_uuid: "8ac23381f96d4614af99bc2e12a4e8d3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38259 } } peers { permanent_uuid: "d0dd96f8ac1743fab87cada20b191272" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42117 } } }
I20260927 12:40:48.496958 2021 tablet_replica.cc:442] T af3111b92c7247058bda7dd4aa5dfca5 P d0dd96f8ac1743fab87cada20b191272: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 12:40:48.497061 2014 tablet_bootstrap.cc:654] T af3111b92c7247058bda7dd4aa5dfca5 P 4b7e8cc171d64b2aa26381c2e2647392: Neither blocks nor log segments found. Creating new log.
I20260927 12:40:48.498242 2014 tablet_bootstrap.cc:492] T af3111b92c7247058bda7dd4aa5dfca5 P 4b7e8cc171d64b2aa26381c2e2647392: No bootstrap required, opened a new log
I20260927 12:40:48.498327 2014 ts_tablet_manager.cc:1397] T af3111b92c7247058bda7dd4aa5dfca5 P 4b7e8cc171d64b2aa26381c2e2647392: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 12:40:48.498710 2014 raft_consensus.cc:348] T af3111b92c7247058bda7dd4aa5dfca5 P 4b7e8cc171d64b2aa26381c2e2647392 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b7e8cc171d64b2aa26381c2e2647392" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43393 } } peers { permanent_uuid: "8ac23381f96d4614af99bc2e12a4e8d3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38259 } } peers { permanent_uuid: "d0dd96f8ac1743fab87cada20b191272" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42117 } }
I20260927 12:40:48.498833 2014 raft_consensus.cc:374] T af3111b92c7247058bda7dd4aa5dfca5 P 4b7e8cc171d64b2aa26381c2e2647392 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:40:48.498876 2014 raft_consensus.cc:729] T af3111b92c7247058bda7dd4aa5dfca5 P 4b7e8cc171d64b2aa26381c2e2647392 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b7e8cc171d64b2aa26381c2e2647392, State: Initialized, Role: FOLLOWER
I20260927 12:40:48.499027 2014 consensus_queue.cc:260] T af3111b92c7247058bda7dd4aa5dfca5 P 4b7e8cc171d64b2aa26381c2e2647392 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b7e8cc171d64b2aa26381c2e2647392" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43393 } } peers { permanent_uuid: "8ac23381f96d4614af99bc2e12a4e8d3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38259 } } peers { permanent_uuid: "d0dd96f8ac1743fab87cada20b191272" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42117 } }
I20260927 12:40:48.499296 2014 ts_tablet_manager.cc:1428] T af3111b92c7247058bda7dd4aa5dfca5 P 4b7e8cc171d64b2aa26381c2e2647392: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 12:40:48.499481 2034 tablet_replica.cc:439] T af3111b92c7247058bda7dd4aa5dfca5 P 4b7e8cc171d64b2aa26381c2e2647392: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b7e8cc171d64b2aa26381c2e2647392" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43393 } } peers { permanent_uuid: "8ac23381f96d4614af99bc2e12a4e8d3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38259 } } peers { permanent_uuid: "d0dd96f8ac1743fab87cada20b191272" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42117 } } }
I20260927 12:40:48.499626 2034 tablet_replica.cc:442] T af3111b92c7247058bda7dd4aa5dfca5 P 4b7e8cc171d64b2aa26381c2e2647392: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 12:40:48.622962 2034 raft_consensus.cc:482] T af3111b92c7247058bda7dd4aa5dfca5 P 4b7e8cc171d64b2aa26381c2e2647392 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:40:48.623184 2034 raft_consensus.cc:504] T af3111b92c7247058bda7dd4aa5dfca5 P 4b7e8cc171d64b2aa26381c2e2647392 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b7e8cc171d64b2aa26381c2e2647392" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43393 } } peers { permanent_uuid: "8ac23381f96d4614af99bc2e12a4e8d3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38259 } } peers { permanent_uuid: "d0dd96f8ac1743fab87cada20b191272" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42117 } }
I20260927 12:40:48.623705 2034 leader_election.cc:288] T af3111b92c7247058bda7dd4aa5dfca5 P 4b7e8cc171d64b2aa26381c2e2647392 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ac23381f96d4614af99bc2e12a4e8d3 (127.0.98.2:38259), d0dd96f8ac1743fab87cada20b191272 (127.0.98.3:42117)
I20260927 12:40:48.624109 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af3111b92c7247058bda7dd4aa5dfca5" candidate_uuid: "4b7e8cc171d64b2aa26381c2e2647392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ac23381f96d4614af99bc2e12a4e8d3" is_pre_election: true
I20260927 12:40:48.624238 1804 raft_consensus.cc:2454] T af3111b92c7247058bda7dd4aa5dfca5 P 8ac23381f96d4614af99bc2e12a4e8d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b7e8cc171d64b2aa26381c2e2647392 in term 0.
I20260927 12:40:48.624185 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af3111b92c7247058bda7dd4aa5dfca5" candidate_uuid: "4b7e8cc171d64b2aa26381c2e2647392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dd96f8ac1743fab87cada20b191272" is_pre_election: true
I20260927 12:40:48.624363 1944 raft_consensus.cc:2454] T af3111b92c7247058bda7dd4aa5dfca5 P d0dd96f8ac1743fab87cada20b191272 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b7e8cc171d64b2aa26381c2e2647392 in term 0.
I20260927 12:40:48.624563 1598 leader_election.cc:302] T af3111b92c7247058bda7dd4aa5dfca5 P 4b7e8cc171d64b2aa26381c2e2647392 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b7e8cc171d64b2aa26381c2e2647392, 8ac23381f96d4614af99bc2e12a4e8d3; no voters:
I20260927 12:40:48.624821 2034 raft_consensus.cc:2781] T af3111b92c7247058bda7dd4aa5dfca5 P 4b7e8cc171d64b2aa26381c2e2647392 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:40:48.624903 2034 raft_consensus.cc:482] T af3111b92c7247058bda7dd4aa5dfca5 P 4b7e8cc171d64b2aa26381c2e2647392 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:40:48.624961 2034 raft_consensus.cc:3037] T af3111b92c7247058bda7dd4aa5dfca5 P 4b7e8cc171d64b2aa26381c2e2647392 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:40:48.626084 2034 raft_consensus.cc:504] T af3111b92c7247058bda7dd4aa5dfca5 P 4b7e8cc171d64b2aa26381c2e2647392 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b7e8cc171d64b2aa26381c2e2647392" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43393 } } peers { permanent_uuid: "8ac23381f96d4614af99bc2e12a4e8d3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38259 } } peers { permanent_uuid: "d0dd96f8ac1743fab87cada20b191272" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42117 } }
I20260927 12:40:48.626438 2034 leader_election.cc:288] T af3111b92c7247058bda7dd4aa5dfca5 P 4b7e8cc171d64b2aa26381c2e2647392 [CANDIDATE]: Term 1 election: Requested vote from peers 8ac23381f96d4614af99bc2e12a4e8d3 (127.0.98.2:38259), d0dd96f8ac1743fab87cada20b191272 (127.0.98.3:42117)
I20260927 12:40:48.626803 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af3111b92c7247058bda7dd4aa5dfca5" candidate_uuid: "4b7e8cc171d64b2aa26381c2e2647392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ac23381f96d4614af99bc2e12a4e8d3"
I20260927 12:40:48.626912 1804 raft_consensus.cc:3037] T af3111b92c7247058bda7dd4aa5dfca5 P 8ac23381f96d4614af99bc2e12a4e8d3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:40:48.626849 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af3111b92c7247058bda7dd4aa5dfca5" candidate_uuid: "4b7e8cc171d64b2aa26381c2e2647392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dd96f8ac1743fab87cada20b191272"
I20260927 12:40:48.627032 1944 raft_consensus.cc:3037] T af3111b92c7247058bda7dd4aa5dfca5 P d0dd96f8ac1743fab87cada20b191272 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:40:48.627810 1804 raft_consensus.cc:2454] T af3111b92c7247058bda7dd4aa5dfca5 P 8ac23381f96d4614af99bc2e12a4e8d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b7e8cc171d64b2aa26381c2e2647392 in term 1.
I20260927 12:40:48.628047 1944 raft_consensus.cc:2454] T af3111b92c7247058bda7dd4aa5dfca5 P d0dd96f8ac1743fab87cada20b191272 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b7e8cc171d64b2aa26381c2e2647392 in term 1.
I20260927 12:40:48.628139 1598 leader_election.cc:302] T af3111b92c7247058bda7dd4aa5dfca5 P 4b7e8cc171d64b2aa26381c2e2647392 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b7e8cc171d64b2aa26381c2e2647392, 8ac23381f96d4614af99bc2e12a4e8d3; no voters:
I20260927 12:40:48.628365 2034 raft_consensus.cc:2781] T af3111b92c7247058bda7dd4aa5dfca5 P 4b7e8cc171d64b2aa26381c2e2647392 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:40:48.628473 2034 raft_consensus.cc:686] T af3111b92c7247058bda7dd4aa5dfca5 P 4b7e8cc171d64b2aa26381c2e2647392 [term 1 LEADER]: Becoming Leader. State: Replica: 4b7e8cc171d64b2aa26381c2e2647392, State: Running, Role: LEADER
I20260927 12:40:48.628628 2034 consensus_queue.cc:237] T af3111b92c7247058bda7dd4aa5dfca5 P 4b7e8cc171d64b2aa26381c2e2647392 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b7e8cc171d64b2aa26381c2e2647392" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43393 } } peers { permanent_uuid: "8ac23381f96d4614af99bc2e12a4e8d3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38259 } } peers { permanent_uuid: "d0dd96f8ac1743fab87cada20b191272" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42117 } }
I20260927 12:40:48.629338 2020 tablet_replica.cc:439] T af3111b92c7247058bda7dd4aa5dfca5 P 4b7e8cc171d64b2aa26381c2e2647392: TxnStatusTablet state changed. Reason: New leader 4b7e8cc171d64b2aa26381c2e2647392. Latest consensus state: current_term: 1 leader_uuid: "4b7e8cc171d64b2aa26381c2e2647392" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b7e8cc171d64b2aa26381c2e2647392" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43393 } } peers { permanent_uuid: "8ac23381f96d4614af99bc2e12a4e8d3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38259 } } peers { permanent_uuid: "d0dd96f8ac1743fab87cada20b191272" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42117 } } }
I20260927 12:40:48.629442 2020 tablet_replica.cc:442] T af3111b92c7247058bda7dd4aa5dfca5 P 4b7e8cc171d64b2aa26381c2e2647392: This TxnStatusTablet replica's current role is: LEADER
I20260927 12:40:48.629750 2056 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 12:40:48.629930 2056 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 12:40:48.630479 1507 catalog_manager.cc:5495] T af3111b92c7247058bda7dd4aa5dfca5 P 4b7e8cc171d64b2aa26381c2e2647392 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b7e8cc171d64b2aa26381c2e2647392 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "4b7e8cc171d64b2aa26381c2e2647392" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b7e8cc171d64b2aa26381c2e2647392" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43393 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ac23381f96d4614af99bc2e12a4e8d3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0dd96f8ac1743fab87cada20b191272" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42117 } health_report { overall_health: UNKNOWN } } }
I20260927 12:40:48.648804 2051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:48.644736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43393 (local address 127.0.0.1:41670)
0927 12:40:48.645009 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0927 12:40:48.645025 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 12:40:48.645158 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:40:48.645445 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:40:48.645448 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:40:48.645464 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 12:40:48.645611 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:48.645618 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:48.646525 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:48.646528 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:48.647518 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:48.647528 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:48.648027 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:48.648030 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:48.648120 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:48.648171 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 12:40:48.648488 (+ 317us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 12:40:48.648501 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 12:40:48.648569 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 12:40:48.648656 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171}
I20260927 12:40:48.648919 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:48.644836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41670 (local address 127.0.98.1:43393)
0927 12:40:48.644980 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:48.644983 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:48.645113 (+ 130us) server_negotiation.cc:407] Connection header received
0927 12:40:48.645276 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:48.645279 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:48.645324 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:48.645394 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:40:48.645838 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:48.646420 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:48.647664 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:48.647915 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:48.648013 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:48.648320 (+ 307us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:40:48.648398 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:40:48.648462 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:48.648731 (+ 269us) server_negotiation.cc:299] Negotiation successful
0927 12:40:48.648790 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 12:40:48.651691 1944 raft_consensus.cc:1261] T af3111b92c7247058bda7dd4aa5dfca5 P d0dd96f8ac1743fab87cada20b191272 [term 1 FOLLOWER]: Refusing update from remote peer 4b7e8cc171d64b2aa26381c2e2647392: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:40:48.651693 1804 raft_consensus.cc:1261] T af3111b92c7247058bda7dd4aa5dfca5 P 8ac23381f96d4614af99bc2e12a4e8d3 [term 1 FOLLOWER]: Refusing update from remote peer 4b7e8cc171d64b2aa26381c2e2647392: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:40:48.652151 2034 consensus_queue.cc:1035] T af3111b92c7247058bda7dd4aa5dfca5 P 4b7e8cc171d64b2aa26381c2e2647392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dd96f8ac1743fab87cada20b191272" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:40:48.652287 2020 consensus_queue.cc:1035] T af3111b92c7247058bda7dd4aa5dfca5 P 4b7e8cc171d64b2aa26381c2e2647392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ac23381f96d4614af99bc2e12a4e8d3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:40:48.653759 2019 tablet_replica.cc:439] T af3111b92c7247058bda7dd4aa5dfca5 P 8ac23381f96d4614af99bc2e12a4e8d3: TxnStatusTablet state changed. Reason: New leader 4b7e8cc171d64b2aa26381c2e2647392. Latest consensus state: current_term: 1 leader_uuid: "4b7e8cc171d64b2aa26381c2e2647392" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b7e8cc171d64b2aa26381c2e2647392" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43393 } } peers { permanent_uuid: "8ac23381f96d4614af99bc2e12a4e8d3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38259 } } peers { permanent_uuid: "d0dd96f8ac1743fab87cada20b191272" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42117 } } }
I20260927 12:40:48.653883 2019 tablet_replica.cc:442] T af3111b92c7247058bda7dd4aa5dfca5 P 8ac23381f96d4614af99bc2e12a4e8d3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 12:40:48.653930 2021 tablet_replica.cc:439] T af3111b92c7247058bda7dd4aa5dfca5 P d0dd96f8ac1743fab87cada20b191272: TxnStatusTablet state changed. Reason: New leader 4b7e8cc171d64b2aa26381c2e2647392. Latest consensus state: current_term: 1 leader_uuid: "4b7e8cc171d64b2aa26381c2e2647392" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b7e8cc171d64b2aa26381c2e2647392" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43393 } } peers { permanent_uuid: "8ac23381f96d4614af99bc2e12a4e8d3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38259 } } peers { permanent_uuid: "d0dd96f8ac1743fab87cada20b191272" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42117 } } }
I20260927 12:40:48.654032 2021 tablet_replica.cc:442] T af3111b92c7247058bda7dd4aa5dfca5 P d0dd96f8ac1743fab87cada20b191272: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 12:40:48.655411 2024 tablet_replica.cc:439] T af3111b92c7247058bda7dd4aa5dfca5 P 4b7e8cc171d64b2aa26381c2e2647392: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4b7e8cc171d64b2aa26381c2e2647392" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b7e8cc171d64b2aa26381c2e2647392" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43393 } } peers { permanent_uuid: "8ac23381f96d4614af99bc2e12a4e8d3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38259 } } peers { permanent_uuid: "d0dd96f8ac1743fab87cada20b191272" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42117 } } }
I20260927 12:40:48.655519 2024 tablet_replica.cc:442] T af3111b92c7247058bda7dd4aa5dfca5 P 4b7e8cc171d64b2aa26381c2e2647392: This TxnStatusTablet replica's current role is: LEADER
I20260927 12:40:48.656172 2020 tablet_replica.cc:439] T af3111b92c7247058bda7dd4aa5dfca5 P 4b7e8cc171d64b2aa26381c2e2647392: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4b7e8cc171d64b2aa26381c2e2647392" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b7e8cc171d64b2aa26381c2e2647392" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43393 } } peers { permanent_uuid: "8ac23381f96d4614af99bc2e12a4e8d3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38259 } } peers { permanent_uuid: "d0dd96f8ac1743fab87cada20b191272" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42117 } } }
I20260927 12:40:48.656263 2020 tablet_replica.cc:442] T af3111b92c7247058bda7dd4aa5dfca5 P 4b7e8cc171d64b2aa26381c2e2647392: This TxnStatusTablet replica's current role is: LEADER
I20260927 12:40:48.657037 2019 tablet_replica.cc:439] T af3111b92c7247058bda7dd4aa5dfca5 P 8ac23381f96d4614af99bc2e12a4e8d3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4b7e8cc171d64b2aa26381c2e2647392" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b7e8cc171d64b2aa26381c2e2647392" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43393 } } peers { permanent_uuid: "8ac23381f96d4614af99bc2e12a4e8d3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38259 } } peers { permanent_uuid: "d0dd96f8ac1743fab87cada20b191272" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42117 } } }
I20260927 12:40:48.657136 2019 tablet_replica.cc:442] T af3111b92c7247058bda7dd4aa5dfca5 P 8ac23381f96d4614af99bc2e12a4e8d3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 12:40:48.657428 2021 tablet_replica.cc:439] T af3111b92c7247058bda7dd4aa5dfca5 P d0dd96f8ac1743fab87cada20b191272: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4b7e8cc171d64b2aa26381c2e2647392" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b7e8cc171d64b2aa26381c2e2647392" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43393 } } peers { permanent_uuid: "8ac23381f96d4614af99bc2e12a4e8d3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38259 } } peers { permanent_uuid: "d0dd96f8ac1743fab87cada20b191272" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42117 } } }
I20260927 12:40:48.657531 2021 tablet_replica.cc:442] T af3111b92c7247058bda7dd4aa5dfca5 P d0dd96f8ac1743fab87cada20b191272: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 12:40:48.659471 2029 mvcc.cc:205] Tried to move back new op lower bound from 7333940628072890368 to 7333940627984453632. Current Snapshot: MvccSnapshot[applied={T|T < 7333940628072890368 or (T in {7333940628072890368})}]
I20260927 12:40:48.659672 2028 mvcc.cc:205] Tried to move back new op lower bound from 7333940628072890368 to 7333940627984453632. Current Snapshot: MvccSnapshot[applied={T|T < 7333940628072890368 or (T in {7333940628072890368})}]
I20260927 12:40:48.664798 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:48.660606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46768 (local address 127.0.98.3:42117)
0927 12:40:48.660706 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:48.660711 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:48.660724 (+ 13us) server_negotiation.cc:407] Connection header received
0927 12:40:48.660792 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:48.660795 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:48.660855 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:48.660995 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:40:48.661267 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:48.662078 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:48.663251 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:48.663560 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:48.663665 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:48.663933 (+ 268us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:40:48.664426 (+ 493us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:40:48.664540 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:48.664609 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 12:40:48.664663 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260927 12:40:48.680003 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:48.671850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54433 (local address 127.0.98.1:43393)
0927 12:40:48.671977 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:48.671982 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:48.676688 (+ 4706us) server_negotiation.cc:407] Connection header received
0927 12:40:48.676877 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:48.676881 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:48.676935 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:48.677016 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:40:48.677272 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:48.678091 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:48.678919 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:48.679254 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:48.679355 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:48.679602 (+ 247us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:40:48.679677 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:40:48.679746 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:48.679823 (+ 77us) server_negotiation.cc:299] Negotiation successful
0927 12:40:48.679874 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 12:40:48.680018 2063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:48.671764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43393 (local address 127.0.98.3:54433)
0927 12:40:48.676604 (+ 4840us) negotiation.cc:107] Waiting for socket to connect
0927 12:40:48.676620 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 12:40:48.676760 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:40:48.677048 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:40:48.677056 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:40:48.677064 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 12:40:48.677166 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:48.677171 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:48.678191 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:48.678194 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:48.678801 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:48.678808 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:48.679359 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:48.679361 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:48.679435 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:48.679475 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 12:40:48.679754 (+ 279us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 12:40:48.679758 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 12:40:48.679807 (+ 49us) client_negotiation.cc:240] Negotiation successful
0927 12:40:48.679869 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4737,"thread_start_us":101,"threads_started":1}
I20260927 12:40:48.687191 1804 raft_consensus.cc:1261] T 4c73652ab07441f6907f8267747f314c P 8ac23381f96d4614af99bc2e12a4e8d3 [term 1 FOLLOWER]: Refusing update from remote peer d0dd96f8ac1743fab87cada20b191272: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:40:48.687417 1664 raft_consensus.cc:1261] T 4c73652ab07441f6907f8267747f314c P 4b7e8cc171d64b2aa26381c2e2647392 [term 1 FOLLOWER]: Refusing update from remote peer d0dd96f8ac1743fab87cada20b191272: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:40:48.687651 2021 consensus_queue.cc:1035] T 4c73652ab07441f6907f8267747f314c P d0dd96f8ac1743fab87cada20b191272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ac23381f96d4614af99bc2e12a4e8d3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:40:48.687973 2044 consensus_queue.cc:1035] T 4c73652ab07441f6907f8267747f314c P d0dd96f8ac1743fab87cada20b191272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b7e8cc171d64b2aa26381c2e2647392" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:40:48.713635 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:48.708131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47631 (local address 127.0.98.3:42117)
0927 12:40:48.708247 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:48.708251 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:48.710070 (+ 1819us) server_negotiation.cc:407] Connection header received
0927 12:40:48.710283 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:48.710287 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:48.710347 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:48.710416 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:40:48.710669 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:48.711177 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:48.712396 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:48.712720 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:48.712841 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:48.713142 (+ 301us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:40:48.713224 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:40:48.713267 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:48.713409 (+ 142us) server_negotiation.cc:299] Negotiation successful
0927 12:40:48.713455 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 12:40:48.713660 2067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:48.708068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42117 (local address 127.0.98.1:47631)
0927 12:40:48.709991 (+ 1923us) negotiation.cc:107] Waiting for socket to connect
0927 12:40:48.710006 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:40:48.710139 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:40:48.710440 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:40:48.710443 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:40:48.710451 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 12:40:48.710568 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:48.710574 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:48.711277 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:48.711281 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:48.712284 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:48.712294 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:48.712844 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:48.712848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:48.712938 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:48.712991 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 12:40:48.713323 (+ 332us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 12:40:48.713328 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 12:40:48.713388 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 12:40:48.713461 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1806,"thread_start_us":105,"threads_started":1}
I20260927 12:40:48.738092 1505 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:40:48.738271 1505 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:40:48.740295 1505 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 299b3d658fec4c4284185eb876ec164a: Sending DeleteTablet for 3 replicas of tablet 4c73652ab07441f6907f8267747f314c
I20260927 12:40:48.740886 1644 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c73652ab07441f6907f8267747f314c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:40:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41656
I20260927 12:40:48.741088 2037 tablet_replica.cc:330] T 4c73652ab07441f6907f8267747f314c P 4b7e8cc171d64b2aa26381c2e2647392: stopping tablet replica
I20260927 12:40:48.741106 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c73652ab07441f6907f8267747f314c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:40:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46762
I20260927 12:40:48.741147 1784 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c73652ab07441f6907f8267747f314c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:40:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33052
I20260927 12:40:48.741191 2037 raft_consensus.cc:2229] T 4c73652ab07441f6907f8267747f314c P 4b7e8cc171d64b2aa26381c2e2647392 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:40:48.741287 2039 tablet_replica.cc:330] T 4c73652ab07441f6907f8267747f314c P d0dd96f8ac1743fab87cada20b191272: stopping tablet replica
I20260927 12:40:48.741315 2038 tablet_replica.cc:330] T 4c73652ab07441f6907f8267747f314c P 8ac23381f96d4614af99bc2e12a4e8d3: stopping tablet replica
I20260927 12:40:48.741307 2037 raft_consensus.cc:2258] T 4c73652ab07441f6907f8267747f314c P 4b7e8cc171d64b2aa26381c2e2647392 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:40:48.741410 2039 raft_consensus.cc:2229] T 4c73652ab07441f6907f8267747f314c P d0dd96f8ac1743fab87cada20b191272 [term 1 LEADER]: Raft consensus shutting down.
I20260927 12:40:48.741421 2038 raft_consensus.cc:2229] T 4c73652ab07441f6907f8267747f314c P 8ac23381f96d4614af99bc2e12a4e8d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:40:48.741547 2038 raft_consensus.cc:2258] T 4c73652ab07441f6907f8267747f314c P 8ac23381f96d4614af99bc2e12a4e8d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:40:48.741732 2039 raft_consensus.cc:2258] T 4c73652ab07441f6907f8267747f314c P d0dd96f8ac1743fab87cada20b191272 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:40:48.742552 2038 ts_tablet_manager.cc:1905] T 4c73652ab07441f6907f8267747f314c P 8ac23381f96d4614af99bc2e12a4e8d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:40:48.742664 2037 ts_tablet_manager.cc:1905] T 4c73652ab07441f6907f8267747f314c P 4b7e8cc171d64b2aa26381c2e2647392: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:40:48.745679 2037 ts_tablet_manager.cc:1918] T 4c73652ab07441f6907f8267747f314c P 4b7e8cc171d64b2aa26381c2e2647392: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 12:40:48.745766 2037 log.cc:1196] T 4c73652ab07441f6907f8267747f314c P 4b7e8cc171d64b2aa26381c2e2647392: Deleting WAL directory at /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/minicluster-data/ts-0/wal/wals/4c73652ab07441f6907f8267747f314c
I20260927 12:40:48.745792 2038 ts_tablet_manager.cc:1918] T 4c73652ab07441f6907f8267747f314c P 8ac23381f96d4614af99bc2e12a4e8d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 12:40:48.745863 2038 log.cc:1196] T 4c73652ab07441f6907f8267747f314c P 8ac23381f96d4614af99bc2e12a4e8d3: Deleting WAL directory at /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/minicluster-data/ts-1/wal/wals/4c73652ab07441f6907f8267747f314c
I20260927 12:40:48.746040 2037 ts_tablet_manager.cc:1939] T 4c73652ab07441f6907f8267747f314c P 4b7e8cc171d64b2aa26381c2e2647392: Deleting consensus metadata
I20260927 12:40:48.746177 2038 ts_tablet_manager.cc:1939] T 4c73652ab07441f6907f8267747f314c P 8ac23381f96d4614af99bc2e12a4e8d3: Deleting consensus metadata
I20260927 12:40:48.746796 1493 catalog_manager.cc:4841] TS 4b7e8cc171d64b2aa26381c2e2647392 (127.0.98.1:43393): tablet 4c73652ab07441f6907f8267747f314c (table test-table [id=30e401904c1240179c1c3a3ec73968f8]) successfully deleted
I20260927 12:40:48.746836 1494 catalog_manager.cc:4841] TS 8ac23381f96d4614af99bc2e12a4e8d3 (127.0.98.2:38259): tablet 4c73652ab07441f6907f8267747f314c (table test-table [id=30e401904c1240179c1c3a3ec73968f8]) successfully deleted
I20260927 12:40:48.747543 2039 ts_tablet_manager.cc:1905] T 4c73652ab07441f6907f8267747f314c P d0dd96f8ac1743fab87cada20b191272: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:40:48.749979 2039 ts_tablet_manager.cc:1918] T 4c73652ab07441f6907f8267747f314c P d0dd96f8ac1743fab87cada20b191272: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 12:40:48.750066 2039 log.cc:1196] T 4c73652ab07441f6907f8267747f314c P d0dd96f8ac1743fab87cada20b191272: Deleting WAL directory at /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790512840836255-392-0/minicluster-data/ts-2/wal/wals/4c73652ab07441f6907f8267747f314c
I20260927 12:40:48.750329 2039 ts_tablet_manager.cc:1939] T 4c73652ab07441f6907f8267747f314c P d0dd96f8ac1743fab87cada20b191272: Deleting consensus metadata
I20260927 12:40:48.751014 1492 catalog_manager.cc:4841] TS d0dd96f8ac1743fab87cada20b191272 (127.0.98.3:42117): tablet 4c73652ab07441f6907f8267747f314c (table test-table [id=30e401904c1240179c1c3a3ec73968f8]) successfully deleted
Sep 27 12:40:48 dist-test-slave-bnc1 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512848, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 12:40:48.759178 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:48.743364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41676 (local address 127.0.98.1:43393)
0927 12:40:48.743486 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:48.743490 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:48.744062 (+ 572us) server_negotiation.cc:407] Connection header received
0927 12:40:48.744260 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:48.744266 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:48.744326 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:48.744381 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:40:48.745610 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:48.746462 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:48.750961 (+ 4499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:48.751269 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:48.751348 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:48.755392 (+ 4044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:40:48.755408 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:40:48.755429 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:40:48.755457 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:40:48.757244 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:48.757760 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:48.757767 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:48.757768 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:48.757866 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:48.758178 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:48.758181 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:48.758183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:48.758448 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:40:48.758683 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:48.758844 (+ 161us) server_negotiation.cc:299] Negotiation successful
0927 12:40:48.758971 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260927 12:40:48.759635 1704 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41676
I20260927 12:40:48.769189 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:48.761329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54992 (local address 127.0.98.62:46813)
0927 12:40:48.761460 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:48.761467 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:48.761958 (+ 491us) server_negotiation.cc:407] Connection header received
0927 12:40:48.762077 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:48.762080 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:48.762161 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:48.762259 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:40:48.762889 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:48.763428 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:48.764226 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:48.764462 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:48.764577 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:48.765967 (+ 1390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:40:48.765971 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:40:48.765973 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:40:48.766001 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:40:48.767534 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:48.767936 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:48.767943 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:48.767946 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:48.768029 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:48.768299 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:48.768304 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:48.768306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:48.768492 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:40:48.768629 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:48.768877 (+ 248us) server_negotiation.cc:299] Negotiation successful
0927 12:40:48.768996 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260927 12:40:48.770066 1505 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54992
I20260927 12:40:48.771710 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 1579
I20260927 12:40:48.781760 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 1719
I20260927 12:40:48.787937 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 1859
I20260927 12:40:48.793430 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 1475
2026-09-27T12:40:48Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2832 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:40:48 dist-test-slave-bnc1 krb5kdc[2077](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:40:48 dist-test-slave-bnc1 krb5kdc[2077](info): set up 2 sockets
Sep 27 12:40:48 dist-test-slave-bnc1 krb5kdc[2077](info): commencing operation
krb5kdc: starting...
W20260927 12:40:49.833305 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/krb5kdc/test-user.keytab.
Sep 27 12:40:49 dist-test-slave-bnc1 krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:40:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:40:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:40:49.952554 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45763
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45763
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:40:50.059082 2090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:40:50.059320 2090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:40:50.059358 2090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:40:50.063377 2090 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:40:50.063434 2090 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 12:40:50.063450 2090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:40:50.063468 2090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:40:50.063531 2090 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:40:50.063560 2090 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:40:50.069267 2090 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45763
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:40:50.070403 2090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:40:50.071453 2090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:40:50.079888 2095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:40:50.079929 2096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:40:50.079883 2098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:40:50.080111 2090 server_base.cc:1034] running on GCE node
I20260927 12:40:50.080874 2090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:40:50.082104 2090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:40:50.083298 2090 hybrid_clock.cc:648] HybridClock initialized: now 1790512850083283 us; error 28 us; skew 500 ppm
Sep 27 12:40:50 dist-test-slave-bnc1 krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512850, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:40:50.086038 2090 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:40:50.086894 2090 webserver.cc:460] Webserver started at http://127.0.98.62:35079/ using document root <none> and password file <none>
I20260927 12:40:50.087399 2090 fs_manager.cc:359] Metadata directory not provided
I20260927 12:40:50.087455 2090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:40:50.087620 2090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:40:50.089648 2090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/minicluster-data/master-0/data/instance:
uuid: "3e9ae20299c2487da31d96a749ba1a33"
format_stamp: "Formatted at 2026-09-27 12:40:50 on dist-test-slave-bnc1"
server_key: "8ea23b58a415563406f47d0ff1030c6f"
server_key_iv: "9feda8b856eeeea83d6227d6bb1d9dbb"
server_key_version: "encryptionkey@0"
I20260927 12:40:50.090178 2090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/minicluster-data/master-0/wal/instance:
uuid: "3e9ae20299c2487da31d96a749ba1a33"
format_stamp: "Formatted at 2026-09-27 12:40:50 on dist-test-slave-bnc1"
server_key: "8ea23b58a415563406f47d0ff1030c6f"
server_key_iv: "9feda8b856eeeea83d6227d6bb1d9dbb"
server_key_version: "encryptionkey@0"
I20260927 12:40:50.093804 2090 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:40:50.096093 2105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:40:50.097427 2090 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:40:50.097532 2090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/minicluster-data/master-0/wal
uuid: "3e9ae20299c2487da31d96a749ba1a33"
format_stamp: "Formatted at 2026-09-27 12:40:50 on dist-test-slave-bnc1"
server_key: "8ea23b58a415563406f47d0ff1030c6f"
server_key_iv: "9feda8b856eeeea83d6227d6bb1d9dbb"
server_key_version: "encryptionkey@0"
I20260927 12:40:50.097704 2090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:40:50.115545 2090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:40:50.118343 2090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:40:50.118499 2090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:40:50.126197 2090 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45763
I20260927 12:40:50.126219 2167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45763 every 8 connection(s)
I20260927 12:40:50.127151 2090 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:40:50.127986 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 2090
I20260927 12:40:50.128073 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/minicluster-data/master-0/wal/instance
I20260927 12:40:50.128348 392 external_mini_cluster.cc:1428] Setting key a48811728e3f7c1e2cde5725db292645
I20260927 12:40:50.130347 2168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:40:50.136655 2168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e9ae20299c2487da31d96a749ba1a33: Bootstrap starting.
Sep 27 12:40:50 dist-test-slave-bnc1 krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:40:50.138964 2168 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e9ae20299c2487da31d96a749ba1a33: Neither blocks nor log segments found. Creating new log.
I20260927 12:40:50.140151 2168 log.cc:824] T 00000000000000000000000000000000 P 3e9ae20299c2487da31d96a749ba1a33: Log is configured to *not* fsync() on all Append() calls
I20260927 12:40:50.142066 2168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e9ae20299c2487da31d96a749ba1a33: No bootstrap required, opened a new log
I20260927 12:40:50.142462 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:50.129785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34778 (local address 127.0.98.62:45763)
0927 12:40:50.130182 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:50.130200 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:50.130243 (+ 43us) server_negotiation.cc:407] Connection header received
0927 12:40:50.130896 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:50.130914 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:50.131238 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:50.131508 (+ 270us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:40:50.132391 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:50.133360 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:50.134526 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:50.134968 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:50.135082 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:50.137873 (+ 2791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:40:50.137878 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:40:50.137889 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:40:50.137914 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:40:50.139641 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:50.140116 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:50.140120 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:50.140124 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:50.140260 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:50.140551 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:50.140555 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:50.140557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:50.140924 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:40:50.141137 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:50.141380 (+ 243us) server_negotiation.cc:299] Negotiation successful
0927 12:40:50.141711 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":140,"threads_started":1}
I20260927 12:40:50.145509 2168 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e9ae20299c2487da31d96a749ba1a33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e9ae20299c2487da31d96a749ba1a33" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45763 } }
I20260927 12:40:50.145797 2168 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e9ae20299c2487da31d96a749ba1a33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:40:50.145884 2168 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e9ae20299c2487da31d96a749ba1a33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e9ae20299c2487da31d96a749ba1a33, State: Initialized, Role: FOLLOWER
I20260927 12:40:50.146333 2168 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e9ae20299c2487da31d96a749ba1a33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e9ae20299c2487da31d96a749ba1a33" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45763 } }
I20260927 12:40:50.146487 2168 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e9ae20299c2487da31d96a749ba1a33 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:40:50.146551 2168 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e9ae20299c2487da31d96a749ba1a33 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:40:50.146675 2168 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e9ae20299c2487da31d96a749ba1a33 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:40:50.147681 2168 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e9ae20299c2487da31d96a749ba1a33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e9ae20299c2487da31d96a749ba1a33" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45763 } }
I20260927 12:40:50.148054 2168 leader_election.cc:302] T 00000000000000000000000000000000 P 3e9ae20299c2487da31d96a749ba1a33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e9ae20299c2487da31d96a749ba1a33; no voters:
I20260927 12:40:50.148391 2168 leader_election.cc:288] T 00000000000000000000000000000000 P 3e9ae20299c2487da31d96a749ba1a33 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:40:50.148609 2173 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e9ae20299c2487da31d96a749ba1a33 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:40:50.148860 2173 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e9ae20299c2487da31d96a749ba1a33 [term 1 LEADER]: Becoming Leader. State: Replica: 3e9ae20299c2487da31d96a749ba1a33, State: Running, Role: LEADER
I20260927 12:40:50.149174 2173 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e9ae20299c2487da31d96a749ba1a33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e9ae20299c2487da31d96a749ba1a33" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45763 } }
I20260927 12:40:50.149716 2168 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e9ae20299c2487da31d96a749ba1a33 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:40:50.150964 2175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e9ae20299c2487da31d96a749ba1a33 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e9ae20299c2487da31d96a749ba1a33. Latest consensus state: current_term: 1 leader_uuid: "3e9ae20299c2487da31d96a749ba1a33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e9ae20299c2487da31d96a749ba1a33" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45763 } } }
I20260927 12:40:50.151012 2174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e9ae20299c2487da31d96a749ba1a33 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e9ae20299c2487da31d96a749ba1a33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e9ae20299c2487da31d96a749ba1a33" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45763 } } }
I20260927 12:40:50.151090 2175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e9ae20299c2487da31d96a749ba1a33 [sys.catalog]: This master's current role is: LEADER
I20260927 12:40:50.151111 2174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e9ae20299c2487da31d96a749ba1a33 [sys.catalog]: This master's current role is: LEADER
I20260927 12:40:50.151463 2182 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:40:50.154626 2182 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:40:50.160118 2182 catalog_manager.cc:1347] Generated new cluster ID: 5c418f29ca614ea2a84e070a4a28c7a8
I20260927 12:40:50.160190 2182 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:40:50.174458 2182 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:40:50.175382 2182 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:40:50.181658 2182 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e9ae20299c2487da31d96a749ba1a33: Generated new TSK 0
I20260927 12:40:50.182338 2182 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 12:40:50.212546 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45763
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:40:50.322281 2194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:40:50.322526 2194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:40:50.322572 2194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:40:50.327101 2194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:40:50.327152 2194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:40:50.327191 2194 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 12:40:50.327211 2194 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 12:40:50.327243 2194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:40:50.332239 2194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:45763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:40:50.333124 2194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:40:50.334072 2194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:40:50.343132 2202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:40:50.343138 2199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:40:50.343142 2200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:40:50.343406 2194 server_base.cc:1034] running on GCE node
I20260927 12:40:50.343873 2194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:40:50.344462 2194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:40:50.345640 2194 hybrid_clock.cc:648] HybridClock initialized: now 1790512850345630 us; error 29 us; skew 500 ppm
Sep 27 12:40:50 dist-test-slave-bnc1 krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512850, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:40:50.348307 2194 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 12:40:50.349138 2194 webserver.cc:460] Webserver started at http://127.0.98.1:42075/ using document root <none> and password file <none>
I20260927 12:40:50.349752 2194 fs_manager.cc:359] Metadata directory not provided
I20260927 12:40:50.349846 2194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:40:50.350121 2194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:40:50.352219 2194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/minicluster-data/ts-0/data/instance:
uuid: "af8e109d5d504a08aea237603b8b23a4"
format_stamp: "Formatted at 2026-09-27 12:40:50 on dist-test-slave-bnc1"
server_key: "cb7fe5f3030c0292a78c8a3b98f4eb7c"
server_key_iv: "7572e80beb5b0c127f0262a59cdd6256"
server_key_version: "encryptionkey@0"
I20260927 12:40:50.352779 2194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/minicluster-data/ts-0/wal/instance:
uuid: "af8e109d5d504a08aea237603b8b23a4"
format_stamp: "Formatted at 2026-09-27 12:40:50 on dist-test-slave-bnc1"
server_key: "cb7fe5f3030c0292a78c8a3b98f4eb7c"
server_key_iv: "7572e80beb5b0c127f0262a59cdd6256"
server_key_version: "encryptionkey@0"
I20260927 12:40:50.356272 2194 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 12:40:50.358547 2209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:40:50.359680 2194 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 12:40:50.359805 2194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/minicluster-data/ts-0/wal
uuid: "af8e109d5d504a08aea237603b8b23a4"
format_stamp: "Formatted at 2026-09-27 12:40:50 on dist-test-slave-bnc1"
server_key: "cb7fe5f3030c0292a78c8a3b98f4eb7c"
server_key_iv: "7572e80beb5b0c127f0262a59cdd6256"
server_key_version: "encryptionkey@0"
I20260927 12:40:50.359903 2194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:40:50.373989 2194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:40:50.377246 2194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:40:50.377449 2194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:40:50.378881 2194 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:40:50.378929 2194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:40:50.378993 2194 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:40:50.379024 2194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 12:40:50 dist-test-slave-bnc1 krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512850, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:40:50.390147 2194 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46773
I20260927 12:40:50.390216 2324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46773 every 8 connection(s)
I20260927 12:40:50.391088 2194 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:40:50.392602 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:50.379754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34471 (local address 127.0.98.62:45763)
0927 12:40:50.379886 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:50.379890 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:50.380908 (+ 1018us) server_negotiation.cc:407] Connection header received
0927 12:40:50.382026 (+ 1118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:50.382033 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:50.382092 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:50.382150 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:40:50.383699 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:50.384240 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:50.385589 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:50.385908 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:50.386041 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:50.388892 (+ 2851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:40:50.388896 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:40:50.388898 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:40:50.388931 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:40:50.390559 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:50.391114 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:50.391117 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:50.391119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:50.391202 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:50.391578 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:50.391581 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:50.391583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:50.391775 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:40:50.391879 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:50.392321 (+ 442us) server_negotiation.cc:299] Negotiation successful
0927 12:40:50.392437 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 12:40:50.393308 2218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:50.380305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45763 (local address 127.0.98.1:34471)
0927 12:40:50.380762 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0927 12:40:50.380800 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 12:40:50.381762 (+ 962us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:40:50.382570 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:40:50.382576 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:40:50.382959 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:40:50.383503 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:50.383521 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:50.384409 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:50.384412 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:50.385449 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:50.385458 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:50.386088 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:50.386092 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:50.386245 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:50.386810 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:40:50.386847 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:40:50.388726 (+ 1879us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:40:50.390671 (+ 1945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:40:50.390686 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:40:50.390702 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:40:50.391005 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:40:50.391283 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:40:50.391286 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:40:50.391288 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:40:50.391465 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:40:50.391874 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:40:50.391880 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:40:50.392104 (+ 224us) client_negotiation.cc:769] Sending connection context
0927 12:40:50.392283 (+ 179us) client_negotiation.cc:240] Negotiation successful
0927 12:40:50.392580 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":132,"threads_started":1}
I20260927 12:40:50.394567 2325 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45763
I20260927 12:40:50.394692 2325 heartbeater.cc:464] Registering TS with master...
I20260927 12:40:50.395205 2325 heartbeater.cc:511] Master 127.0.98.62:45763 requested a full tablet report, sending...
I20260927 12:40:50.396801 2121 ts_manager.cc:194] Registered new tserver with Master: af8e109d5d504a08aea237603b8b23a4 (127.0.98.1:46773)
I20260927 12:40:50.398177 2121 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34471
I20260927 12:40:50.399006 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 2194
I20260927 12:40:50.399111 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/minicluster-data/ts-0/wal/instance
I20260927 12:40:50.399421 392 external_mini_cluster.cc:1428] Setting key e155cfd9292628b88da6a011b2dec156
I20260927 12:40:50.404865 2331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:50.396643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45763 (local address 127.0.98.1:57567)
0927 12:40:50.397735 (+ 1092us) negotiation.cc:107] Waiting for socket to connect
0927 12:40:50.397747 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 12:40:50.397836 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:40:50.398069 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:40:50.398071 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:40:50.398219 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:40:50.398335 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:50.398342 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:50.399299 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:50.399302 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:50.400077 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:50.400086 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:50.400628 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:50.400631 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:50.400718 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:50.401248 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:40:50.401259 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:40:50.401654 (+ 395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:40:50.403370 (+ 1716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:40:50.403375 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:40:50.403379 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:40:50.403638 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:40:50.403960 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:40:50.403963 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:40:50.403964 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:40:50.404036 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:40:50.404465 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:40:50.404467 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:40:50.404534 (+ 67us) client_negotiation.cc:769] Sending connection context
0927 12:40:50.404613 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 12:40:50.404710 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1025,"thread_start_us":101,"threads_started":1}
I20260927 12:40:50.405061 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:50.396707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57567 (local address 127.0.98.62:45763)
0927 12:40:50.396814 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:50.396817 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:50.397818 (+ 1001us) server_negotiation.cc:407] Connection header received
0927 12:40:50.397922 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:50.397925 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:50.397966 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:50.398051 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:40:50.398452 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:50.399165 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:50.400169 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:50.400499 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:50.400593 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:50.401799 (+ 1206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:40:50.401806 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:40:50.401812 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:40:50.401844 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:40:50.403259 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:50.403771 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:50.403777 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:50.403781 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:50.403869 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:50.404125 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:50.404131 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:50.404136 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:50.404372 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:40:50.404476 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:50.404698 (+ 222us) server_negotiation.cc:299] Negotiation successful
0927 12:40:50.404839 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 12:40:50.424649 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45763
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:40:50.552074 2334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:40:50.552379 2334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:40:50.552469 2334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:40:50.556586 2334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:40:50.556680 2334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:40:50.556748 2334 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 12:40:50.556792 2334 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 12:40:50.556860 2334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:40:50.562294 2334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:45763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2334
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:40:50.563400 2334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:40:50.564431 2334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 12:40:50.573921 2334 server_base.cc:1034] running on GCE node
W20260927 12:40:50.573938 2339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:40:50.574167 2342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:40:50.573933 2340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:40:50.574563 2334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:40:50.575162 2334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:40:50.576342 2334 hybrid_clock.cc:648] HybridClock initialized: now 1790512850576327 us; error 30 us; skew 500 ppm
Sep 27 12:40:50 dist-test-slave-bnc1 krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512850, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:40:50.579063 2334 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 12:40:50.579963 2334 webserver.cc:460] Webserver started at http://127.0.98.2:46479/ using document root <none> and password file <none>
I20260927 12:40:50.580513 2334 fs_manager.cc:359] Metadata directory not provided
I20260927 12:40:50.580602 2334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:40:50.580828 2334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:40:50.582937 2334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/minicluster-data/ts-1/data/instance:
uuid: "6092fc28ff124c22aef46193200cd07c"
format_stamp: "Formatted at 2026-09-27 12:40:50 on dist-test-slave-bnc1"
server_key: "2ea9725f8b51df26b2a438951ee78cc5"
server_key_iv: "e2ab752f29091f1ebf44e3f338f0532a"
server_key_version: "encryptionkey@0"
I20260927 12:40:50.583523 2334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/minicluster-data/ts-1/wal/instance:
uuid: "6092fc28ff124c22aef46193200cd07c"
format_stamp: "Formatted at 2026-09-27 12:40:50 on dist-test-slave-bnc1"
server_key: "2ea9725f8b51df26b2a438951ee78cc5"
server_key_iv: "e2ab752f29091f1ebf44e3f338f0532a"
server_key_version: "encryptionkey@0"
I20260927 12:40:50.587222 2334 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 12:40:50.589697 2349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:40:50.590926 2334 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:40:50.591068 2334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/minicluster-data/ts-1/wal
uuid: "6092fc28ff124c22aef46193200cd07c"
format_stamp: "Formatted at 2026-09-27 12:40:50 on dist-test-slave-bnc1"
server_key: "2ea9725f8b51df26b2a438951ee78cc5"
server_key_iv: "e2ab752f29091f1ebf44e3f338f0532a"
server_key_version: "encryptionkey@0"
I20260927 12:40:50.591189 2334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:40:50.619376 2334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:40:50.622462 2334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:40:50.622658 2334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:40:50.624050 2334 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:40:50.624122 2334 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:40:50.624195 2334 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:40:50.624243 2334 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:40:50.634660 2334 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40853
I20260927 12:40:50.634717 2464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40853 every 8 connection(s)
Sep 27 12:40:50 dist-test-slave-bnc1 krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512850, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:40:50.635567 2334 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:40:50.639333 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:50.624963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45183 (local address 127.0.98.62:45763)
0927 12:40:50.625088 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:50.625092 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:50.626266 (+ 1174us) server_negotiation.cc:407] Connection header received
0927 12:40:50.627493 (+ 1227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:50.627497 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:50.627555 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:50.627654 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:40:50.629327 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:50.630235 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:50.632046 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:50.632256 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:50.632358 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:50.635612 (+ 3254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:40:50.635616 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:40:50.635619 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:40:50.635651 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:40:50.637191 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:50.637719 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:50.637722 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:50.637724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:50.637825 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:50.638225 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:50.638231 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:50.638234 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:50.638509 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:40:50.638615 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:50.639118 (+ 503us) server_negotiation.cc:299] Negotiation successful
0927 12:40:50.639216 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 12:40:50.640132 2358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:50.625524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45763 (local address 127.0.98.2:45183)
0927 12:40:50.626113 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0927 12:40:50.626158 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 12:40:50.627190 (+ 1032us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:40:50.627884 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:40:50.627896 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:40:50.628395 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:40:50.629146 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:50.629169 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:50.630376 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:50.630381 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:50.631929 (+ 1548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:50.631940 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:50.632413 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:50.632419 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:50.632663 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:50.633811 (+ 1148us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:40:50.633827 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:40:50.635435 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:40:50.637290 (+ 1855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:40:50.637294 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:40:50.637305 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:40:50.637568 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:40:50.637929 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:40:50.637932 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:40:50.637934 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:40:50.638104 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:40:50.638615 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:40:50.638621 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:40:50.638900 (+ 279us) client_negotiation.cc:769] Sending connection context
0927 12:40:50.639047 (+ 147us) client_negotiation.cc:240] Negotiation successful
0927 12:40:50.639279 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":161,"threads_started":1}
I20260927 12:40:50.640705 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 2334
I20260927 12:40:50.640841 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/minicluster-data/ts-1/wal/instance
I20260927 12:40:50.641157 392 external_mini_cluster.cc:1428] Setting key 04835875a17bf50c988e12bf34cda6ef
I20260927 12:40:50.641848 2465 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45763
I20260927 12:40:50.642004 2465 heartbeater.cc:464] Registering TS with master...
I20260927 12:40:50.642725 2465 heartbeater.cc:511] Master 127.0.98.62:45763 requested a full tablet report, sending...
I20260927 12:40:50.643784 2122 ts_manager.cc:194] Registered new tserver with Master: 6092fc28ff124c22aef46193200cd07c (127.0.98.2:40853)
I20260927 12:40:50.644575 2122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45183
I20260927 12:40:50.651032 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:50.643834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46585 (local address 127.0.98.62:45763)
0927 12:40:50.643937 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:50.643940 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:50.644194 (+ 254us) server_negotiation.cc:407] Connection header received
0927 12:40:50.644291 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:50.644293 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:50.644332 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:50.644396 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:40:50.644894 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:50.645412 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:50.646362 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:50.646588 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:50.646694 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:50.647845 (+ 1151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:40:50.647849 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:40:50.647851 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:40:50.647881 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:40:50.649269 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:50.649788 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:50.649793 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:50.649795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:50.649875 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:50.650148 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:50.650152 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:50.650154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:50.650394 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:40:50.650483 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:50.650773 (+ 290us) server_negotiation.cc:299] Negotiation successful
0927 12:40:50.650894 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 12:40:50.651046 2472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:50.643787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45763 (local address 127.0.98.2:46585)
0927 12:40:50.644103 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0927 12:40:50.644117 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:40:50.644211 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:40:50.644430 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:40:50.644433 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:40:50.644638 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:40:50.644749 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:50.644756 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:50.645534 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:50.645537 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:50.646229 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:50.646237 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:50.646695 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:50.646699 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:50.646788 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:50.647289 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:40:50.647315 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:40:50.647694 (+ 379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:40:50.649403 (+ 1709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:40:50.649406 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:40:50.649408 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:40:50.649694 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:40:50.649967 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:40:50.649971 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:40:50.649974 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:40:50.650059 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:40:50.650488 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:40:50.650491 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:40:50.650584 (+ 93us) client_negotiation.cc:769] Sending connection context
0927 12:40:50.650709 (+ 125us) client_negotiation.cc:240] Negotiation successful
0927 12:40:50.650843 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"mutex_wait_us":41,"thread_start_us":148,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 12:40:50.668408 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45763
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:40:50.777948 2474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:40:50.778223 2474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:40:50.778306 2474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:40:50.782234 2474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:40:50.782313 2474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:40:50.782382 2474 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 12:40:50.782424 2474 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 12:40:50.782492 2474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:40:50.787662 2474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:45763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:40:50.788717 2474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:40:50.789786 2474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:40:50.799510 2480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:40:50.799460 2482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:40:50.799520 2474 server_base.cc:1034] running on GCE node
W20260927 12:40:50.799464 2479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:40:50.800093 2474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:40:50.800778 2474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:40:50.801972 2474 hybrid_clock.cc:648] HybridClock initialized: now 1790512850801950 us; error 44 us; skew 500 ppm
Sep 27 12:40:50 dist-test-slave-bnc1 krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512850, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:40:50.805718 2474 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 12:40:50.806614 2474 webserver.cc:460] Webserver started at http://127.0.98.3:39349/ using document root <none> and password file <none>
I20260927 12:40:50.807113 2474 fs_manager.cc:359] Metadata directory not provided
I20260927 12:40:50.807176 2474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:40:50.807330 2474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:40:50.809286 2474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/minicluster-data/ts-2/data/instance:
uuid: "10b5013c5e0a4050ad9447ea19df245c"
format_stamp: "Formatted at 2026-09-27 12:40:50 on dist-test-slave-bnc1"
server_key: "011557de5ad872978d31e048eb88be71"
server_key_iv: "a7791f48eba1749f4b6e857ba9241419"
server_key_version: "encryptionkey@0"
I20260927 12:40:50.809885 2474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/minicluster-data/ts-2/wal/instance:
uuid: "10b5013c5e0a4050ad9447ea19df245c"
format_stamp: "Formatted at 2026-09-27 12:40:50 on dist-test-slave-bnc1"
server_key: "011557de5ad872978d31e048eb88be71"
server_key_iv: "a7791f48eba1749f4b6e857ba9241419"
server_key_version: "encryptionkey@0"
I20260927 12:40:50.813519 2474 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 12:40:50.815851 2489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:40:50.817073 2474 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:40:50.817183 2474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/minicluster-data/ts-2/wal
uuid: "10b5013c5e0a4050ad9447ea19df245c"
format_stamp: "Formatted at 2026-09-27 12:40:50 on dist-test-slave-bnc1"
server_key: "011557de5ad872978d31e048eb88be71"
server_key_iv: "a7791f48eba1749f4b6e857ba9241419"
server_key_version: "encryptionkey@0"
I20260927 12:40:50.817281 2474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:40:50.834501 2474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:40:50.837976 2474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:40:50.838186 2474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:40:50.839618 2474 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:40:50.839666 2474 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:40:50.839735 2474 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:40:50.839766 2474 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 12:40:50 dist-test-slave-bnc1 krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512850, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:40:50.850780 2474 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33869
I20260927 12:40:50.850916 2604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33869 every 8 connection(s)
I20260927 12:40:50.851922 2474 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:40:50.854125 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 2474
I20260927 12:40:50.854259 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/minicluster-data/ts-2/wal/instance
I20260927 12:40:50.854337 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:50.840610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48641 (local address 127.0.98.62:45763)
0927 12:40:50.840737 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:50.840740 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:50.841871 (+ 1131us) server_negotiation.cc:407] Connection header received
0927 12:40:50.843025 (+ 1154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:50.843028 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:50.843084 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:50.843195 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:40:50.844738 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:50.845658 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:50.846817 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:50.847052 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:50.847166 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:50.850509 (+ 3343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:40:50.850513 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:40:50.850519 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:40:50.850557 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:40:50.852138 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:50.852747 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:50.852750 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:50.852752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:50.852834 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:50.853182 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:50.853185 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:50.853187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:50.853377 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:40:50.853477 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:50.854044 (+ 567us) server_negotiation.cc:299] Negotiation successful
0927 12:40:50.854176 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 12:40:50.854626 392 external_mini_cluster.cc:1428] Setting key 2b3f7df470f258bda71bca62c1a2945b
I20260927 12:40:50.855237 2498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:50.841167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45763 (local address 127.0.98.3:48641)
0927 12:40:50.841698 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0927 12:40:50.841772 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0927 12:40:50.842748 (+ 976us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:40:50.843370 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:40:50.843379 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:40:50.843891 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:40:50.844594 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:50.844617 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:50.845778 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:50.845781 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:50.846678 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:50.846687 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:50.847212 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:50.847216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:50.847415 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:50.848517 (+ 1102us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:40:50.848533 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:40:50.850310 (+ 1777us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:40:50.852261 (+ 1951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:40:50.852267 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:40:50.852281 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:40:50.852627 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:40:50.852937 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:40:50.852941 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:40:50.852943 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:40:50.853079 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:40:50.853488 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:40:50.853496 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:40:50.853805 (+ 309us) client_negotiation.cc:769] Sending connection context
0927 12:40:50.854004 (+ 199us) client_negotiation.cc:240] Negotiation successful
0927 12:40:50.854319 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":143,"threads_started":1}
I20260927 12:40:50.856472 2605 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45763
I20260927 12:40:50.856613 2605 heartbeater.cc:464] Registering TS with master...
I20260927 12:40:50.857097 2605 heartbeater.cc:511] Master 127.0.98.62:45763 requested a full tablet report, sending...
I20260927 12:40:50.858127 2121 ts_manager.cc:194] Registered new tserver with Master: 10b5013c5e0a4050ad9447ea19df245c (127.0.98.3:33869)
I20260927 12:40:50.858851 2121 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48641
I20260927 12:40:50.866178 2611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:50.859060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45763 (local address 127.0.98.3:42191)
0927 12:40:50.859392 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0927 12:40:50.859406 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:40:50.859528 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:40:50.859767 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:40:50.859770 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:40:50.859967 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:40:50.860056 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:50.860061 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:50.860786 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:50.860789 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:50.861346 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:50.861353 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:50.861898 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:50.861902 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:50.861996 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:50.862525 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:40:50.862544 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:40:50.862868 (+ 324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:40:50.864687 (+ 1819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:40:50.864692 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:40:50.864694 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:40:50.864888 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:40:50.865156 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:40:50.865160 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:40:50.865162 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:40:50.865245 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:40:50.865651 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:40:50.865654 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:40:50.865748 (+ 94us) client_negotiation.cc:769] Sending connection context
0927 12:40:50.865852 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 12:40:50.865987 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":92,"threads_started":1}
I20260927 12:40:50.866173 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:50.858807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42191 (local address 127.0.98.62:45763)
0927 12:40:50.858940 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:50.858947 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:50.859489 (+ 542us) server_negotiation.cc:407] Connection header received
0927 12:40:50.859604 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:50.859607 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:50.859647 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:50.859739 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:40:50.860166 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:50.860667 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:50.861470 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:50.861788 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:50.861907 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:50.863002 (+ 1095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:40:50.863006 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:40:50.863009 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:40:50.863031 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:40:50.864584 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:50.864981 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:50.864987 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:50.864989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:50.865068 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:50.865333 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:50.865339 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:50.865340 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:50.865507 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:40:50.865661 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:50.865921 (+ 260us) server_negotiation.cc:299] Negotiation successful
0927 12:40:50.866021 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 12:40:50.868607 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 12:40:50 dist-test-slave-bnc1 krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512850, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 12:40:50 dist-test-slave-bnc1 krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512850, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:40:50.893548 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:50.885285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34790 (local address 127.0.98.62:45763)
0927 12:40:50.885437 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:50.885441 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:50.885454 (+ 13us) server_negotiation.cc:407] Connection header received
0927 12:40:50.885495 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:50.885497 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:50.885534 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:50.885676 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:40:50.886409 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:50.886925 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:50.887756 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:50.887999 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:50.888107 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:50.890559 (+ 2452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:40:50.890563 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:40:50.890566 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:40:50.890597 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:40:50.892008 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:50.892522 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:50.892525 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:50.892527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:50.892607 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:50.892858 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:50.892860 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:50.892861 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:50.893030 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:40:50.893131 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:50.893322 (+ 191us) server_negotiation.cc:299] Negotiation successful
0927 12:40:50.893415 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 12:40:50.896673 2121 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34790:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 12:40:50.899787 2121 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:40:50.913969 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:50.908254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47786 (local address 127.0.98.3:33869)
0927 12:40:50.908809 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:50.908821 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:50.909132 (+ 311us) server_negotiation.cc:407] Connection header received
0927 12:40:50.909199 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:50.909204 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:50.909351 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:50.909503 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:40:50.909882 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:50.910728 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:50.912394 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:50.913243 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:50.913368 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:50.913544 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:50.913706 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 12:40:50.913806 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":104,"threads_started":1}
I20260927 12:40:50.913957 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:50.907993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33869 (local address 127.0.0.1:47786)
0927 12:40:50.908953 (+ 960us) negotiation.cc:107] Waiting for socket to connect
0927 12:40:50.908967 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:40:50.909045 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:40:50.909515 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:40:50.909522 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:40:50.909547 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:40:50.909731 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:50.909743 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:50.910862 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:50.910865 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:50.912279 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:50.912287 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:50.913368 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:50.913372 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:50.913485 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:50.913525 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 12:40:50.913665 (+ 140us) client_negotiation.cc:240] Negotiation successful
0927 12:40:50.913786 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":892,"mutex_wait_us":30,"thread_start_us":83,"threads_started":1}
I20260927 12:40:50.914278 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:50.908048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42600 (local address 127.0.98.1:46773)
0927 12:40:50.908435 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:50.908442 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:50.909025 (+ 583us) server_negotiation.cc:407] Connection header received
0927 12:40:50.909132 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:50.909138 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:50.909352 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:50.909503 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:40:50.909963 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:50.911168 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:50.912561 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:50.913280 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:50.913374 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:50.913519 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:50.914042 (+ 523us) server_negotiation.cc:299] Negotiation successful
0927 12:40:50.914152 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":69,"threads_started":1}
I20260927 12:40:50.914474 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:50.909165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40853 (local address 127.0.0.1:54582)
0927 12:40:50.909771 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0927 12:40:50.909786 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:40:50.909864 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:40:50.910554 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:40:50.910557 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:40:50.910575 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:40:50.910664 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:50.910669 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:50.912158 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:50.912161 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:50.912989 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:50.912996 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:50.914047 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:50.914051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:50.914167 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:50.914194 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 12:40:50.914249 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 12:40:50.914312 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":90,"threads_started":1}
I20260927 12:40:50.914522 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:50.907936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46773 (local address 127.0.0.1:42600)
0927 12:40:50.908685 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0927 12:40:50.908737 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 12:40:50.908874 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:40:50.909691 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:40:50.909694 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:40:50.909713 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:40:50.909831 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:50.909838 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:50.911274 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:50.911280 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:50.912437 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:50.912446 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:50.913673 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:50.913676 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:50.913778 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:50.913801 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 12:40:50.913874 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 12:40:50.913936 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"mutex_wait_us":2,"thread_start_us":98,"threads_started":1}
I20260927 12:40:50.915266 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:50.909626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54582 (local address 127.0.98.2:40853)
0927 12:40:50.910098 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:50.910108 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:50.910129 (+ 21us) server_negotiation.cc:407] Connection header received
0927 12:40:50.910197 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:50.910202 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:50.910388 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:50.910543 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:40:50.910908 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:50.912043 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:50.913110 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:50.913912 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:50.914799 (+ 887us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:50.914970 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:50.915020 (+ 50us) server_negotiation.cc:299] Negotiation successful
0927 12:40:50.915108 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":116,"threads_started":1}
I20260927 12:40:50.917285 2539 tablet_service.cc:1467] Processing CreateTablet for tablet 297b382dc1144ef791adad96d115cacf (DEFAULT_TABLE table=test-table [id=2835283a5c7946f4b2fc127a97f44a49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:40:50.917275 2259 tablet_service.cc:1467] Processing CreateTablet for tablet 297b382dc1144ef791adad96d115cacf (DEFAULT_TABLE table=test-table [id=2835283a5c7946f4b2fc127a97f44a49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:40:50.918448 2539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 297b382dc1144ef791adad96d115cacf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:40:50.918443 2259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 297b382dc1144ef791adad96d115cacf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:40:50.919888 2399 tablet_service.cc:1467] Processing CreateTablet for tablet 297b382dc1144ef791adad96d115cacf (DEFAULT_TABLE table=test-table [id=2835283a5c7946f4b2fc127a97f44a49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:40:50.920889 2399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 297b382dc1144ef791adad96d115cacf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:40:50.926277 2629 tablet_bootstrap.cc:492] T 297b382dc1144ef791adad96d115cacf P 10b5013c5e0a4050ad9447ea19df245c: Bootstrap starting.
I20260927 12:40:50.927160 2628 tablet_bootstrap.cc:492] T 297b382dc1144ef791adad96d115cacf P af8e109d5d504a08aea237603b8b23a4: Bootstrap starting.
I20260927 12:40:50.927764 2630 tablet_bootstrap.cc:492] T 297b382dc1144ef791adad96d115cacf P 6092fc28ff124c22aef46193200cd07c: Bootstrap starting.
I20260927 12:40:50.928608 2629 tablet_bootstrap.cc:654] T 297b382dc1144ef791adad96d115cacf P 10b5013c5e0a4050ad9447ea19df245c: Neither blocks nor log segments found. Creating new log.
I20260927 12:40:50.929042 2628 tablet_bootstrap.cc:654] T 297b382dc1144ef791adad96d115cacf P af8e109d5d504a08aea237603b8b23a4: Neither blocks nor log segments found. Creating new log.
I20260927 12:40:50.929457 2629 log.cc:824] T 297b382dc1144ef791adad96d115cacf P 10b5013c5e0a4050ad9447ea19df245c: Log is configured to *not* fsync() on all Append() calls
I20260927 12:40:50.929697 2630 tablet_bootstrap.cc:654] T 297b382dc1144ef791adad96d115cacf P 6092fc28ff124c22aef46193200cd07c: Neither blocks nor log segments found. Creating new log.
I20260927 12:40:50.929865 2628 log.cc:824] T 297b382dc1144ef791adad96d115cacf P af8e109d5d504a08aea237603b8b23a4: Log is configured to *not* fsync() on all Append() calls
I20260927 12:40:50.930492 2630 log.cc:824] T 297b382dc1144ef791adad96d115cacf P 6092fc28ff124c22aef46193200cd07c: Log is configured to *not* fsync() on all Append() calls
I20260927 12:40:50.931537 2629 tablet_bootstrap.cc:492] T 297b382dc1144ef791adad96d115cacf P 10b5013c5e0a4050ad9447ea19df245c: No bootstrap required, opened a new log
I20260927 12:40:50.931680 2628 tablet_bootstrap.cc:492] T 297b382dc1144ef791adad96d115cacf P af8e109d5d504a08aea237603b8b23a4: No bootstrap required, opened a new log
I20260927 12:40:50.931730 2629 ts_tablet_manager.cc:1397] T 297b382dc1144ef791adad96d115cacf P 10b5013c5e0a4050ad9447ea19df245c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 12:40:50.931835 2628 ts_tablet_manager.cc:1397] T 297b382dc1144ef791adad96d115cacf P af8e109d5d504a08aea237603b8b23a4: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 12:40:50.932307 2630 tablet_bootstrap.cc:492] T 297b382dc1144ef791adad96d115cacf P 6092fc28ff124c22aef46193200cd07c: No bootstrap required, opened a new log
I20260927 12:40:50.932494 2630 ts_tablet_manager.cc:1397] T 297b382dc1144ef791adad96d115cacf P 6092fc28ff124c22aef46193200cd07c: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260927 12:40:50.935329 2630 raft_consensus.cc:348] T 297b382dc1144ef791adad96d115cacf P 6092fc28ff124c22aef46193200cd07c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8e109d5d504a08aea237603b8b23a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46773 } } peers { permanent_uuid: "10b5013c5e0a4050ad9447ea19df245c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33869 } } peers { permanent_uuid: "6092fc28ff124c22aef46193200cd07c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40853 } }
I20260927 12:40:50.935559 2630 raft_consensus.cc:374] T 297b382dc1144ef791adad96d115cacf P 6092fc28ff124c22aef46193200cd07c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:40:50.935611 2630 raft_consensus.cc:729] T 297b382dc1144ef791adad96d115cacf P 6092fc28ff124c22aef46193200cd07c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6092fc28ff124c22aef46193200cd07c, State: Initialized, Role: FOLLOWER
I20260927 12:40:50.935774 2629 raft_consensus.cc:348] T 297b382dc1144ef791adad96d115cacf P 10b5013c5e0a4050ad9447ea19df245c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8e109d5d504a08aea237603b8b23a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46773 } } peers { permanent_uuid: "10b5013c5e0a4050ad9447ea19df245c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33869 } } peers { permanent_uuid: "6092fc28ff124c22aef46193200cd07c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40853 } }
I20260927 12:40:50.936020 2629 raft_consensus.cc:374] T 297b382dc1144ef791adad96d115cacf P 10b5013c5e0a4050ad9447ea19df245c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:40:50.936075 2629 raft_consensus.cc:729] T 297b382dc1144ef791adad96d115cacf P 10b5013c5e0a4050ad9447ea19df245c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b5013c5e0a4050ad9447ea19df245c, State: Initialized, Role: FOLLOWER
I20260927 12:40:50.936058 2630 consensus_queue.cc:260] T 297b382dc1144ef791adad96d115cacf P 6092fc28ff124c22aef46193200cd07c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8e109d5d504a08aea237603b8b23a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46773 } } peers { permanent_uuid: "10b5013c5e0a4050ad9447ea19df245c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33869 } } peers { permanent_uuid: "6092fc28ff124c22aef46193200cd07c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40853 } }
I20260927 12:40:50.936219 2628 raft_consensus.cc:348] T 297b382dc1144ef791adad96d115cacf P af8e109d5d504a08aea237603b8b23a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8e109d5d504a08aea237603b8b23a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46773 } } peers { permanent_uuid: "10b5013c5e0a4050ad9447ea19df245c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33869 } } peers { permanent_uuid: "6092fc28ff124c22aef46193200cd07c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40853 } }
I20260927 12:40:50.936450 2628 raft_consensus.cc:374] T 297b382dc1144ef791adad96d115cacf P af8e109d5d504a08aea237603b8b23a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:40:50.936513 2628 raft_consensus.cc:729] T 297b382dc1144ef791adad96d115cacf P af8e109d5d504a08aea237603b8b23a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af8e109d5d504a08aea237603b8b23a4, State: Initialized, Role: FOLLOWER
I20260927 12:40:50.936537 2629 consensus_queue.cc:260] T 297b382dc1144ef791adad96d115cacf P 10b5013c5e0a4050ad9447ea19df245c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8e109d5d504a08aea237603b8b23a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46773 } } peers { permanent_uuid: "10b5013c5e0a4050ad9447ea19df245c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33869 } } peers { permanent_uuid: "6092fc28ff124c22aef46193200cd07c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40853 } }
I20260927 12:40:50.936851 2465 heartbeater.cc:503] Master 127.0.98.62:45763 was elected leader, sending a full tablet report...
I20260927 12:40:50.936955 2628 consensus_queue.cc:260] T 297b382dc1144ef791adad96d115cacf P af8e109d5d504a08aea237603b8b23a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8e109d5d504a08aea237603b8b23a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46773 } } peers { permanent_uuid: "10b5013c5e0a4050ad9447ea19df245c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33869 } } peers { permanent_uuid: "6092fc28ff124c22aef46193200cd07c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40853 } }
I20260927 12:40:50.937491 2629 ts_tablet_manager.cc:1428] T 297b382dc1144ef791adad96d115cacf P 10b5013c5e0a4050ad9447ea19df245c: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 12:40:50.937721 2605 heartbeater.cc:503] Master 127.0.98.62:45763 was elected leader, sending a full tablet report...
I20260927 12:40:50.937840 2628 ts_tablet_manager.cc:1428] T 297b382dc1144ef791adad96d115cacf P af8e109d5d504a08aea237603b8b23a4: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 12:40:50.937883 2325 heartbeater.cc:503] Master 127.0.98.62:45763 was elected leader, sending a full tablet report...
I20260927 12:40:50.938951 2630 ts_tablet_manager.cc:1428] T 297b382dc1144ef791adad96d115cacf P 6092fc28ff124c22aef46193200cd07c: Time spent starting tablet: real 0.006s user 0.004s sys 0.001s
I20260927 12:40:51.053195 2634 raft_consensus.cc:482] T 297b382dc1144ef791adad96d115cacf P 6092fc28ff124c22aef46193200cd07c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:40:51.053444 2634 raft_consensus.cc:504] T 297b382dc1144ef791adad96d115cacf P 6092fc28ff124c22aef46193200cd07c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8e109d5d504a08aea237603b8b23a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46773 } } peers { permanent_uuid: "10b5013c5e0a4050ad9447ea19df245c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33869 } } peers { permanent_uuid: "6092fc28ff124c22aef46193200cd07c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40853 } }
I20260927 12:40:51.054610 2634 leader_election.cc:288] T 297b382dc1144ef791adad96d115cacf P 6092fc28ff124c22aef46193200cd07c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af8e109d5d504a08aea237603b8b23a4 (127.0.98.1:46773), 10b5013c5e0a4050ad9447ea19df245c (127.0.98.3:33869)
I20260927 12:40:51.058688 2358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:51.054968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33869 (local address 127.0.98.2:38875)
0927 12:40:51.055127 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 12:40:51.055143 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 12:40:51.055252 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:40:51.055535 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:40:51.055537 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:40:51.055555 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:40:51.055684 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:51.055689 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:51.056462 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:51.056468 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:51.057342 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:51.057350 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:51.058250 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:51.058254 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:51.058368 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:51.058390 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 12:40:51.058465 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 12:40:51.058541 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"spinlock_wait_cycles":1280}
I20260927 12:40:51.059046 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:51.055030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38875 (local address 127.0.98.3:33869)
0927 12:40:51.055145 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:51.055150 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:51.055199 (+ 49us) server_negotiation.cc:407] Connection header received
0927 12:40:51.055385 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:51.055388 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:51.055433 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:51.055523 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:40:51.055796 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:51.056356 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:51.057485 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:51.058115 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:51.058240 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:51.058305 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:51.058853 (+ 548us) server_negotiation.cc:299] Negotiation successful
0927 12:40:51.058928 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 12:40:51.059530 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:51.054972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46773 (local address 127.0.98.2:41445)
0927 12:40:51.055441 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0927 12:40:51.055452 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 12:40:51.055588 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:40:51.055854 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:40:51.055857 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:40:51.055872 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:40:51.056325 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:51.056331 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:51.057156 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:51.057162 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:51.058218 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:51.058227 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:51.059126 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:51.059132 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:51.059251 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:51.059277 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 12:40:51.059352 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 12:40:51.059418 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"mutex_wait_us":120,"spinlock_wait_cycles":2304,"thread_start_us":165,"threads_started":1}
I20260927 12:40:51.059590 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:51.055013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41445 (local address 127.0.98.1:46773)
0927 12:40:51.055128 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:51.055133 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:51.055533 (+ 400us) server_negotiation.cc:407] Connection header received
0927 12:40:51.055704 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:51.055707 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:51.055758 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:51.055834 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:40:51.056437 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:51.057051 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:51.058374 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:51.058989 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:51.059113 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:51.059178 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:51.059391 (+ 213us) server_negotiation.cc:299] Negotiation successful
0927 12:40:51.059471 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 12:40:51.059654 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297b382dc1144ef791adad96d115cacf" candidate_uuid: "6092fc28ff124c22aef46193200cd07c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b5013c5e0a4050ad9447ea19df245c" is_pre_election: true
I20260927 12:40:51.059984 2559 raft_consensus.cc:2454] T 297b382dc1144ef791adad96d115cacf P 10b5013c5e0a4050ad9447ea19df245c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6092fc28ff124c22aef46193200cd07c in term 0.
I20260927 12:40:51.060276 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297b382dc1144ef791adad96d115cacf" candidate_uuid: "6092fc28ff124c22aef46193200cd07c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af8e109d5d504a08aea237603b8b23a4" is_pre_election: true
I20260927 12:40:51.060580 2279 raft_consensus.cc:2454] T 297b382dc1144ef791adad96d115cacf P af8e109d5d504a08aea237603b8b23a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6092fc28ff124c22aef46193200cd07c in term 0.
I20260927 12:40:51.060566 2351 leader_election.cc:302] T 297b382dc1144ef791adad96d115cacf P 6092fc28ff124c22aef46193200cd07c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b5013c5e0a4050ad9447ea19df245c, 6092fc28ff124c22aef46193200cd07c; no voters:
I20260927 12:40:51.060880 2634 raft_consensus.cc:2781] T 297b382dc1144ef791adad96d115cacf P 6092fc28ff124c22aef46193200cd07c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:40:51.060977 2634 raft_consensus.cc:482] T 297b382dc1144ef791adad96d115cacf P 6092fc28ff124c22aef46193200cd07c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:40:51.061025 2634 raft_consensus.cc:3037] T 297b382dc1144ef791adad96d115cacf P 6092fc28ff124c22aef46193200cd07c [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:40:51.062101 2634 raft_consensus.cc:504] T 297b382dc1144ef791adad96d115cacf P 6092fc28ff124c22aef46193200cd07c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8e109d5d504a08aea237603b8b23a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46773 } } peers { permanent_uuid: "10b5013c5e0a4050ad9447ea19df245c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33869 } } peers { permanent_uuid: "6092fc28ff124c22aef46193200cd07c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40853 } }
I20260927 12:40:51.062448 2634 leader_election.cc:288] T 297b382dc1144ef791adad96d115cacf P 6092fc28ff124c22aef46193200cd07c [CANDIDATE]: Term 1 election: Requested vote from peers af8e109d5d504a08aea237603b8b23a4 (127.0.98.1:46773), 10b5013c5e0a4050ad9447ea19df245c (127.0.98.3:33869)
I20260927 12:40:51.062742 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297b382dc1144ef791adad96d115cacf" candidate_uuid: "6092fc28ff124c22aef46193200cd07c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af8e109d5d504a08aea237603b8b23a4"
I20260927 12:40:51.062851 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297b382dc1144ef791adad96d115cacf" candidate_uuid: "6092fc28ff124c22aef46193200cd07c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b5013c5e0a4050ad9447ea19df245c"
I20260927 12:40:51.062860 2279 raft_consensus.cc:3037] T 297b382dc1144ef791adad96d115cacf P af8e109d5d504a08aea237603b8b23a4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:40:51.062983 2559 raft_consensus.cc:3037] T 297b382dc1144ef791adad96d115cacf P 10b5013c5e0a4050ad9447ea19df245c [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:40:51.064008 2279 raft_consensus.cc:2454] T 297b382dc1144ef791adad96d115cacf P af8e109d5d504a08aea237603b8b23a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6092fc28ff124c22aef46193200cd07c in term 1.
I20260927 12:40:51.064139 2559 raft_consensus.cc:2454] T 297b382dc1144ef791adad96d115cacf P 10b5013c5e0a4050ad9447ea19df245c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6092fc28ff124c22aef46193200cd07c in term 1.
I20260927 12:40:51.064325 2353 leader_election.cc:302] T 297b382dc1144ef791adad96d115cacf P 6092fc28ff124c22aef46193200cd07c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6092fc28ff124c22aef46193200cd07c, af8e109d5d504a08aea237603b8b23a4; no voters:
I20260927 12:40:51.064504 2634 raft_consensus.cc:2781] T 297b382dc1144ef791adad96d115cacf P 6092fc28ff124c22aef46193200cd07c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:40:51.064739 2634 raft_consensus.cc:686] T 297b382dc1144ef791adad96d115cacf P 6092fc28ff124c22aef46193200cd07c [term 1 LEADER]: Becoming Leader. State: Replica: 6092fc28ff124c22aef46193200cd07c, State: Running, Role: LEADER
I20260927 12:40:51.065059 2634 consensus_queue.cc:237] T 297b382dc1144ef791adad96d115cacf P 6092fc28ff124c22aef46193200cd07c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8e109d5d504a08aea237603b8b23a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46773 } } peers { permanent_uuid: "10b5013c5e0a4050ad9447ea19df245c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33869 } } peers { permanent_uuid: "6092fc28ff124c22aef46193200cd07c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40853 } }
I20260927 12:40:51.068111 2121 catalog_manager.cc:5495] T 297b382dc1144ef791adad96d115cacf P 6092fc28ff124c22aef46193200cd07c reported cstate change: term changed from 0 to 1, leader changed from <none> to 6092fc28ff124c22aef46193200cd07c (127.0.98.2). New cstate: current_term: 1 leader_uuid: "6092fc28ff124c22aef46193200cd07c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8e109d5d504a08aea237603b8b23a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10b5013c5e0a4050ad9447ea19df245c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6092fc28ff124c22aef46193200cd07c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40853 } health_report { overall_health: HEALTHY } } }
W20260927 12:40:51.103159 2606 tablet.cc:2369] T 297b382dc1144ef791adad96d115cacf P 10b5013c5e0a4050ad9447ea19df245c: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 12:40:51 dist-test-slave-bnc1 krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512850, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:40:51.123314 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:51.115282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34794 (local address 127.0.98.62:45763)
0927 12:40:51.115438 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:51.115444 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:51.115596 (+ 152us) server_negotiation.cc:407] Connection header received
0927 12:40:51.115758 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:51.115761 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:51.115804 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:51.115886 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:40:51.116543 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:51.117045 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:51.117852 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:51.118056 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:51.118170 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:51.120262 (+ 2092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:40:51.120266 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:40:51.120270 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:40:51.120294 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:40:51.121698 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:51.122117 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:51.122123 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:51.122125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:51.122204 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:51.122480 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:51.122483 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:51.122484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:51.122640 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:40:51.122747 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:51.123025 (+ 278us) server_negotiation.cc:299] Negotiation successful
0927 12:40:51.123164 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":76,"server-negotiator.queue_time_us":49}
I20260927 12:40:51.123314 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:51.115181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45763 (local address 127.0.0.1:34794)
0927 12:40:51.115510 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0927 12:40:51.115521 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 12:40:51.115647 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:40:51.115902 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:40:51.115904 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:40:51.116331 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:40:51.116417 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:51.116424 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:51.117156 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:51.117159 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:51.117732 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:51.117739 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:51.118167 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:51.118170 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:51.118314 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:51.118802 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:40:51.118813 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:40:51.120134 (+ 1321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:40:51.121795 (+ 1661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:40:51.121806 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:40:51.121812 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:40:51.122023 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:40:51.122292 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:40:51.122297 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:40:51.122299 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:40:51.122389 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:40:51.122745 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:40:51.122752 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:40:51.122863 (+ 111us) client_negotiation.cc:769] Sending connection context
0927 12:40:51.122973 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 12:40:51.123118 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"mutex_wait_us":47,"thread_start_us":115,"threads_started":1}
I20260927 12:40:51.127491 2121 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34794:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 12:40:51.128289 2121 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:40:51.133486 2259 tablet_service.cc:1467] Processing CreateTablet for tablet ef65749ca29a48b98b53f34e3c86d39b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0c3c1c6886d945609938e0cf294e9265]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 12:40:51.133482 2399 tablet_service.cc:1467] Processing CreateTablet for tablet ef65749ca29a48b98b53f34e3c86d39b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0c3c1c6886d945609938e0cf294e9265]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 12:40:51.134227 2399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef65749ca29a48b98b53f34e3c86d39b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:40:51.134227 2259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef65749ca29a48b98b53f34e3c86d39b. 1 dirs total, 0 dirs full, 0 dirs failed
W20260927 12:40:51.136664 2466 tablet.cc:2369] T 297b382dc1144ef791adad96d115cacf P 6092fc28ff124c22aef46193200cd07c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:40:51.137079 2628 tablet_bootstrap.cc:492] T ef65749ca29a48b98b53f34e3c86d39b P af8e109d5d504a08aea237603b8b23a4: Bootstrap starting.
I20260927 12:40:51.137550 2539 tablet_service.cc:1467] Processing CreateTablet for tablet ef65749ca29a48b98b53f34e3c86d39b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0c3c1c6886d945609938e0cf294e9265]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 12:40:51.138068 2628 tablet_bootstrap.cc:654] T ef65749ca29a48b98b53f34e3c86d39b P af8e109d5d504a08aea237603b8b23a4: Neither blocks nor log segments found. Creating new log.
I20260927 12:40:51.138252 2539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef65749ca29a48b98b53f34e3c86d39b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:40:51.138938 2630 tablet_bootstrap.cc:492] T ef65749ca29a48b98b53f34e3c86d39b P 6092fc28ff124c22aef46193200cd07c: Bootstrap starting.
I20260927 12:40:51.139935 2630 tablet_bootstrap.cc:654] T ef65749ca29a48b98b53f34e3c86d39b P 6092fc28ff124c22aef46193200cd07c: Neither blocks nor log segments found. Creating new log.
I20260927 12:40:51.141328 2629 tablet_bootstrap.cc:492] T ef65749ca29a48b98b53f34e3c86d39b P 10b5013c5e0a4050ad9447ea19df245c: Bootstrap starting.
I20260927 12:40:51.141332 2628 tablet_bootstrap.cc:492] T ef65749ca29a48b98b53f34e3c86d39b P af8e109d5d504a08aea237603b8b23a4: No bootstrap required, opened a new log
I20260927 12:40:51.141419 2628 ts_tablet_manager.cc:1397] T ef65749ca29a48b98b53f34e3c86d39b P af8e109d5d504a08aea237603b8b23a4: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260927 12:40:51.141448 2630 tablet_bootstrap.cc:492] T ef65749ca29a48b98b53f34e3c86d39b P 6092fc28ff124c22aef46193200cd07c: No bootstrap required, opened a new log
I20260927 12:40:51.141516 2630 ts_tablet_manager.cc:1397] T ef65749ca29a48b98b53f34e3c86d39b P 6092fc28ff124c22aef46193200cd07c: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260927 12:40:51.141942 2628 raft_consensus.cc:348] T ef65749ca29a48b98b53f34e3c86d39b P af8e109d5d504a08aea237603b8b23a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8e109d5d504a08aea237603b8b23a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46773 } } peers { permanent_uuid: "6092fc28ff124c22aef46193200cd07c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40853 } } peers { permanent_uuid: "10b5013c5e0a4050ad9447ea19df245c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33869 } }
I20260927 12:40:51.142066 2628 raft_consensus.cc:374] T ef65749ca29a48b98b53f34e3c86d39b P af8e109d5d504a08aea237603b8b23a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:40:51.142005 2630 raft_consensus.cc:348] T ef65749ca29a48b98b53f34e3c86d39b P 6092fc28ff124c22aef46193200cd07c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8e109d5d504a08aea237603b8b23a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46773 } } peers { permanent_uuid: "6092fc28ff124c22aef46193200cd07c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40853 } } peers { permanent_uuid: "10b5013c5e0a4050ad9447ea19df245c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33869 } }
I20260927 12:40:51.142099 2628 raft_consensus.cc:729] T ef65749ca29a48b98b53f34e3c86d39b P af8e109d5d504a08aea237603b8b23a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af8e109d5d504a08aea237603b8b23a4, State: Initialized, Role: FOLLOWER
I20260927 12:40:51.142107 2630 raft_consensus.cc:374] T ef65749ca29a48b98b53f34e3c86d39b P 6092fc28ff124c22aef46193200cd07c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:40:51.142140 2630 raft_consensus.cc:729] T ef65749ca29a48b98b53f34e3c86d39b P 6092fc28ff124c22aef46193200cd07c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6092fc28ff124c22aef46193200cd07c, State: Initialized, Role: FOLLOWER
I20260927 12:40:51.142243 2628 consensus_queue.cc:260] T ef65749ca29a48b98b53f34e3c86d39b P af8e109d5d504a08aea237603b8b23a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8e109d5d504a08aea237603b8b23a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46773 } } peers { permanent_uuid: "6092fc28ff124c22aef46193200cd07c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40853 } } peers { permanent_uuid: "10b5013c5e0a4050ad9447ea19df245c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33869 } }
I20260927 12:40:51.142319 2629 tablet_bootstrap.cc:654] T ef65749ca29a48b98b53f34e3c86d39b P 10b5013c5e0a4050ad9447ea19df245c: Neither blocks nor log segments found. Creating new log.
I20260927 12:40:51.142280 2630 consensus_queue.cc:260] T ef65749ca29a48b98b53f34e3c86d39b P 6092fc28ff124c22aef46193200cd07c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8e109d5d504a08aea237603b8b23a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46773 } } peers { permanent_uuid: "6092fc28ff124c22aef46193200cd07c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40853 } } peers { permanent_uuid: "10b5013c5e0a4050ad9447ea19df245c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33869 } }
I20260927 12:40:51.142539 2628 ts_tablet_manager.cc:1428] T ef65749ca29a48b98b53f34e3c86d39b P af8e109d5d504a08aea237603b8b23a4: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 12:40:51.142556 2630 ts_tablet_manager.cc:1428] T ef65749ca29a48b98b53f34e3c86d39b P 6092fc28ff124c22aef46193200cd07c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 12:40:51.142498 2636 tablet_replica.cc:439] T ef65749ca29a48b98b53f34e3c86d39b P af8e109d5d504a08aea237603b8b23a4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8e109d5d504a08aea237603b8b23a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46773 } } peers { permanent_uuid: "6092fc28ff124c22aef46193200cd07c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40853 } } peers { permanent_uuid: "10b5013c5e0a4050ad9447ea19df245c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33869 } } }
I20260927 12:40:51.142642 2636 tablet_replica.cc:442] T ef65749ca29a48b98b53f34e3c86d39b P af8e109d5d504a08aea237603b8b23a4: This TxnStatusTablet replica's current role is: FOLLOWER
W20260927 12:40:51.142755 2326 tablet.cc:2369] T 297b382dc1144ef791adad96d115cacf P af8e109d5d504a08aea237603b8b23a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:40:51.143316 2634 tablet_replica.cc:439] T ef65749ca29a48b98b53f34e3c86d39b P 6092fc28ff124c22aef46193200cd07c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8e109d5d504a08aea237603b8b23a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46773 } } peers { permanent_uuid: "6092fc28ff124c22aef46193200cd07c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40853 } } peers { permanent_uuid: "10b5013c5e0a4050ad9447ea19df245c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33869 } } }
I20260927 12:40:51.143448 2634 tablet_replica.cc:442] T ef65749ca29a48b98b53f34e3c86d39b P 6092fc28ff124c22aef46193200cd07c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 12:40:51.144438 2629 tablet_bootstrap.cc:492] T ef65749ca29a48b98b53f34e3c86d39b P 10b5013c5e0a4050ad9447ea19df245c: No bootstrap required, opened a new log
I20260927 12:40:51.144511 2629 ts_tablet_manager.cc:1397] T ef65749ca29a48b98b53f34e3c86d39b P 10b5013c5e0a4050ad9447ea19df245c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 12:40:51.144991 2629 raft_consensus.cc:348] T ef65749ca29a48b98b53f34e3c86d39b P 10b5013c5e0a4050ad9447ea19df245c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8e109d5d504a08aea237603b8b23a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46773 } } peers { permanent_uuid: "6092fc28ff124c22aef46193200cd07c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40853 } } peers { permanent_uuid: "10b5013c5e0a4050ad9447ea19df245c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33869 } }
I20260927 12:40:51.145081 2629 raft_consensus.cc:374] T ef65749ca29a48b98b53f34e3c86d39b P 10b5013c5e0a4050ad9447ea19df245c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:40:51.145104 2629 raft_consensus.cc:729] T ef65749ca29a48b98b53f34e3c86d39b P 10b5013c5e0a4050ad9447ea19df245c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b5013c5e0a4050ad9447ea19df245c, State: Initialized, Role: FOLLOWER
I20260927 12:40:51.145251 2629 consensus_queue.cc:260] T ef65749ca29a48b98b53f34e3c86d39b P 10b5013c5e0a4050ad9447ea19df245c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8e109d5d504a08aea237603b8b23a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46773 } } peers { permanent_uuid: "6092fc28ff124c22aef46193200cd07c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40853 } } peers { permanent_uuid: "10b5013c5e0a4050ad9447ea19df245c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33869 } }
I20260927 12:40:51.145489 2629 ts_tablet_manager.cc:1428] T ef65749ca29a48b98b53f34e3c86d39b P 10b5013c5e0a4050ad9447ea19df245c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 12:40:51.145517 2635 tablet_replica.cc:439] T ef65749ca29a48b98b53f34e3c86d39b P 10b5013c5e0a4050ad9447ea19df245c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8e109d5d504a08aea237603b8b23a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46773 } } peers { permanent_uuid: "6092fc28ff124c22aef46193200cd07c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40853 } } peers { permanent_uuid: "10b5013c5e0a4050ad9447ea19df245c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33869 } } }
I20260927 12:40:51.145673 2635 tablet_replica.cc:442] T ef65749ca29a48b98b53f34e3c86d39b P 10b5013c5e0a4050ad9447ea19df245c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 12:40:51.249887 2634 raft_consensus.cc:482] T ef65749ca29a48b98b53f34e3c86d39b P 6092fc28ff124c22aef46193200cd07c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:40:51.250051 2634 raft_consensus.cc:504] T ef65749ca29a48b98b53f34e3c86d39b P 6092fc28ff124c22aef46193200cd07c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8e109d5d504a08aea237603b8b23a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46773 } } peers { permanent_uuid: "6092fc28ff124c22aef46193200cd07c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40853 } } peers { permanent_uuid: "10b5013c5e0a4050ad9447ea19df245c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33869 } }
I20260927 12:40:51.250597 2634 leader_election.cc:288] T ef65749ca29a48b98b53f34e3c86d39b P 6092fc28ff124c22aef46193200cd07c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af8e109d5d504a08aea237603b8b23a4 (127.0.98.1:46773), 10b5013c5e0a4050ad9447ea19df245c (127.0.98.3:33869)
I20260927 12:40:51.251047 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef65749ca29a48b98b53f34e3c86d39b" candidate_uuid: "6092fc28ff124c22aef46193200cd07c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af8e109d5d504a08aea237603b8b23a4" is_pre_election: true
I20260927 12:40:51.251056 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef65749ca29a48b98b53f34e3c86d39b" candidate_uuid: "6092fc28ff124c22aef46193200cd07c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b5013c5e0a4050ad9447ea19df245c" is_pre_election: true
I20260927 12:40:51.251204 2279 raft_consensus.cc:2454] T ef65749ca29a48b98b53f34e3c86d39b P af8e109d5d504a08aea237603b8b23a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6092fc28ff124c22aef46193200cd07c in term 0.
I20260927 12:40:51.251215 2559 raft_consensus.cc:2454] T ef65749ca29a48b98b53f34e3c86d39b P 10b5013c5e0a4050ad9447ea19df245c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6092fc28ff124c22aef46193200cd07c in term 0.
I20260927 12:40:51.251505 2353 leader_election.cc:302] T ef65749ca29a48b98b53f34e3c86d39b P 6092fc28ff124c22aef46193200cd07c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6092fc28ff124c22aef46193200cd07c, af8e109d5d504a08aea237603b8b23a4; no voters:
I20260927 12:40:51.251677 2634 raft_consensus.cc:2781] T ef65749ca29a48b98b53f34e3c86d39b P 6092fc28ff124c22aef46193200cd07c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:40:51.251735 2634 raft_consensus.cc:482] T ef65749ca29a48b98b53f34e3c86d39b P 6092fc28ff124c22aef46193200cd07c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:40:51.251783 2634 raft_consensus.cc:3037] T ef65749ca29a48b98b53f34e3c86d39b P 6092fc28ff124c22aef46193200cd07c [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:40:51.252754 2634 raft_consensus.cc:504] T ef65749ca29a48b98b53f34e3c86d39b P 6092fc28ff124c22aef46193200cd07c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8e109d5d504a08aea237603b8b23a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46773 } } peers { permanent_uuid: "6092fc28ff124c22aef46193200cd07c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40853 } } peers { permanent_uuid: "10b5013c5e0a4050ad9447ea19df245c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33869 } }
I20260927 12:40:51.253122 2634 leader_election.cc:288] T ef65749ca29a48b98b53f34e3c86d39b P 6092fc28ff124c22aef46193200cd07c [CANDIDATE]: Term 1 election: Requested vote from peers af8e109d5d504a08aea237603b8b23a4 (127.0.98.1:46773), 10b5013c5e0a4050ad9447ea19df245c (127.0.98.3:33869)
I20260927 12:40:51.253449 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef65749ca29a48b98b53f34e3c86d39b" candidate_uuid: "6092fc28ff124c22aef46193200cd07c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af8e109d5d504a08aea237603b8b23a4"
I20260927 12:40:51.253547 2279 raft_consensus.cc:3037] T ef65749ca29a48b98b53f34e3c86d39b P af8e109d5d504a08aea237603b8b23a4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:40:51.253509 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef65749ca29a48b98b53f34e3c86d39b" candidate_uuid: "6092fc28ff124c22aef46193200cd07c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b5013c5e0a4050ad9447ea19df245c"
I20260927 12:40:51.253676 2559 raft_consensus.cc:3037] T ef65749ca29a48b98b53f34e3c86d39b P 10b5013c5e0a4050ad9447ea19df245c [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:40:51.254706 2279 raft_consensus.cc:2454] T ef65749ca29a48b98b53f34e3c86d39b P af8e109d5d504a08aea237603b8b23a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6092fc28ff124c22aef46193200cd07c in term 1.
I20260927 12:40:51.254820 2559 raft_consensus.cc:2454] T ef65749ca29a48b98b53f34e3c86d39b P 10b5013c5e0a4050ad9447ea19df245c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6092fc28ff124c22aef46193200cd07c in term 1.
I20260927 12:40:51.254971 2353 leader_election.cc:302] T ef65749ca29a48b98b53f34e3c86d39b P 6092fc28ff124c22aef46193200cd07c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6092fc28ff124c22aef46193200cd07c, af8e109d5d504a08aea237603b8b23a4; no voters:
I20260927 12:40:51.255122 2634 raft_consensus.cc:2781] T ef65749ca29a48b98b53f34e3c86d39b P 6092fc28ff124c22aef46193200cd07c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:40:51.255208 2634 raft_consensus.cc:686] T ef65749ca29a48b98b53f34e3c86d39b P 6092fc28ff124c22aef46193200cd07c [term 1 LEADER]: Becoming Leader. State: Replica: 6092fc28ff124c22aef46193200cd07c, State: Running, Role: LEADER
I20260927 12:40:51.255347 2634 consensus_queue.cc:237] T ef65749ca29a48b98b53f34e3c86d39b P 6092fc28ff124c22aef46193200cd07c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8e109d5d504a08aea237603b8b23a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46773 } } peers { permanent_uuid: "6092fc28ff124c22aef46193200cd07c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40853 } } peers { permanent_uuid: "10b5013c5e0a4050ad9447ea19df245c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33869 } }
I20260927 12:40:51.255970 2638 tablet_replica.cc:439] T ef65749ca29a48b98b53f34e3c86d39b P 6092fc28ff124c22aef46193200cd07c: TxnStatusTablet state changed. Reason: New leader 6092fc28ff124c22aef46193200cd07c. Latest consensus state: current_term: 1 leader_uuid: "6092fc28ff124c22aef46193200cd07c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8e109d5d504a08aea237603b8b23a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46773 } } peers { permanent_uuid: "6092fc28ff124c22aef46193200cd07c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40853 } } peers { permanent_uuid: "10b5013c5e0a4050ad9447ea19df245c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33869 } } }
I20260927 12:40:51.256090 2638 tablet_replica.cc:442] T ef65749ca29a48b98b53f34e3c86d39b P 6092fc28ff124c22aef46193200cd07c: This TxnStatusTablet replica's current role is: LEADER
I20260927 12:40:51.256356 2650 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 12:40:51.256587 2650 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 12:40:51.256963 2122 catalog_manager.cc:5495] T ef65749ca29a48b98b53f34e3c86d39b P 6092fc28ff124c22aef46193200cd07c reported cstate change: term changed from 0 to 1, leader changed from <none> to 6092fc28ff124c22aef46193200cd07c (127.0.98.2). New cstate: current_term: 1 leader_uuid: "6092fc28ff124c22aef46193200cd07c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8e109d5d504a08aea237603b8b23a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6092fc28ff124c22aef46193200cd07c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10b5013c5e0a4050ad9447ea19df245c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20260927 12:40:51.294132 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:51.289797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40853 (local address 127.0.0.1:54586)
0927 12:40:51.289937 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 12:40:51.289960 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 12:40:51.290043 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:40:51.290364 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:40:51.290367 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:40:51.290373 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 12:40:51.290462 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:51.290467 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:51.291487 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:51.291490 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:51.292087 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:51.292094 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:51.292716 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:51.292753 (+ 37us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:51.292940 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:51.293003 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 12:40:51.293846 (+ 843us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 12:40:51.293859 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 12:40:51.293919 (+ 60us) client_negotiation.cc:240] Negotiation successful
0927 12:40:51.294026 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260927 12:40:51.294149 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:51.289906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54586 (local address 127.0.98.2:40853)
0927 12:40:51.290030 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:51.290035 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:51.290050 (+ 15us) server_negotiation.cc:407] Connection header received
0927 12:40:51.290167 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:51.290170 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:51.290235 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:51.290349 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:40:51.290570 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:51.291373 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:51.292208 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:51.292574 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:51.292676 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:51.293176 (+ 500us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:40:51.293750 (+ 574us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:40:51.293863 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:51.293934 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 12:40:51.293992 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 12:40:51.299551 2559 raft_consensus.cc:1261] T ef65749ca29a48b98b53f34e3c86d39b P 10b5013c5e0a4050ad9447ea19df245c [term 1 FOLLOWER]: Refusing update from remote peer 6092fc28ff124c22aef46193200cd07c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:40:51.299593 2279 raft_consensus.cc:1261] T ef65749ca29a48b98b53f34e3c86d39b P af8e109d5d504a08aea237603b8b23a4 [term 1 FOLLOWER]: Refusing update from remote peer 6092fc28ff124c22aef46193200cd07c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:40:51.300235 2634 consensus_queue.cc:1035] T ef65749ca29a48b98b53f34e3c86d39b P 6092fc28ff124c22aef46193200cd07c [LEADER]: Connected to new peer: Peer: permanent_uuid: "af8e109d5d504a08aea237603b8b23a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:40:51.300391 2638 consensus_queue.cc:1035] T ef65749ca29a48b98b53f34e3c86d39b P 6092fc28ff124c22aef46193200cd07c [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b5013c5e0a4050ad9447ea19df245c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:40:51.305228 2636 tablet_replica.cc:439] T ef65749ca29a48b98b53f34e3c86d39b P af8e109d5d504a08aea237603b8b23a4: TxnStatusTablet state changed. Reason: New leader 6092fc28ff124c22aef46193200cd07c. Latest consensus state: current_term: 1 leader_uuid: "6092fc28ff124c22aef46193200cd07c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8e109d5d504a08aea237603b8b23a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46773 } } peers { permanent_uuid: "6092fc28ff124c22aef46193200cd07c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40853 } } peers { permanent_uuid: "10b5013c5e0a4050ad9447ea19df245c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33869 } } }
I20260927 12:40:51.305382 2636 tablet_replica.cc:442] T ef65749ca29a48b98b53f34e3c86d39b P af8e109d5d504a08aea237603b8b23a4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 12:40:51.309182 2635 tablet_replica.cc:439] T ef65749ca29a48b98b53f34e3c86d39b P 10b5013c5e0a4050ad9447ea19df245c: TxnStatusTablet state changed. Reason: New leader 6092fc28ff124c22aef46193200cd07c. Latest consensus state: current_term: 1 leader_uuid: "6092fc28ff124c22aef46193200cd07c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8e109d5d504a08aea237603b8b23a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46773 } } peers { permanent_uuid: "6092fc28ff124c22aef46193200cd07c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40853 } } peers { permanent_uuid: "10b5013c5e0a4050ad9447ea19df245c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33869 } } }
I20260927 12:40:51.309315 2635 tablet_replica.cc:442] T ef65749ca29a48b98b53f34e3c86d39b P 10b5013c5e0a4050ad9447ea19df245c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 12:40:51.310489 2634 tablet_replica.cc:439] T ef65749ca29a48b98b53f34e3c86d39b P 6092fc28ff124c22aef46193200cd07c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6092fc28ff124c22aef46193200cd07c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8e109d5d504a08aea237603b8b23a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46773 } } peers { permanent_uuid: "6092fc28ff124c22aef46193200cd07c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40853 } } peers { permanent_uuid: "10b5013c5e0a4050ad9447ea19df245c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33869 } } }
I20260927 12:40:51.310598 2634 tablet_replica.cc:442] T ef65749ca29a48b98b53f34e3c86d39b P 6092fc28ff124c22aef46193200cd07c: This TxnStatusTablet replica's current role is: LEADER
I20260927 12:40:51.310806 2653 mvcc.cc:205] Tried to move back new op lower bound from 7333940638915727360 to 7333940638742867968. Current Snapshot: MvccSnapshot[applied={T|T < 7333940638915727360}]
I20260927 12:40:51.311784 2638 tablet_replica.cc:439] T ef65749ca29a48b98b53f34e3c86d39b P 6092fc28ff124c22aef46193200cd07c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6092fc28ff124c22aef46193200cd07c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8e109d5d504a08aea237603b8b23a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46773 } } peers { permanent_uuid: "6092fc28ff124c22aef46193200cd07c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40853 } } peers { permanent_uuid: "10b5013c5e0a4050ad9447ea19df245c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33869 } } }
I20260927 12:40:51.311901 2638 tablet_replica.cc:442] T ef65749ca29a48b98b53f34e3c86d39b P 6092fc28ff124c22aef46193200cd07c: This TxnStatusTablet replica's current role is: LEADER
I20260927 12:40:51.312222 2654 mvcc.cc:205] Tried to move back new op lower bound from 7333940638915727360 to 7333940638742867968. Current Snapshot: MvccSnapshot[applied={T|T < 7333940638915727360}]
I20260927 12:40:51.312415 2636 tablet_replica.cc:439] T ef65749ca29a48b98b53f34e3c86d39b P af8e109d5d504a08aea237603b8b23a4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6092fc28ff124c22aef46193200cd07c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8e109d5d504a08aea237603b8b23a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46773 } } peers { permanent_uuid: "6092fc28ff124c22aef46193200cd07c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40853 } } peers { permanent_uuid: "10b5013c5e0a4050ad9447ea19df245c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33869 } } }
I20260927 12:40:51.312520 2636 tablet_replica.cc:442] T ef65749ca29a48b98b53f34e3c86d39b P af8e109d5d504a08aea237603b8b23a4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 12:40:51.313285 2635 tablet_replica.cc:439] T ef65749ca29a48b98b53f34e3c86d39b P 10b5013c5e0a4050ad9447ea19df245c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6092fc28ff124c22aef46193200cd07c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8e109d5d504a08aea237603b8b23a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46773 } } peers { permanent_uuid: "6092fc28ff124c22aef46193200cd07c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40853 } } peers { permanent_uuid: "10b5013c5e0a4050ad9447ea19df245c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33869 } } }
I20260927 12:40:51.313402 2635 tablet_replica.cc:442] T ef65749ca29a48b98b53f34e3c86d39b P 10b5013c5e0a4050ad9447ea19df245c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 12:40:51.314579 2656 mvcc.cc:205] Tried to move back new op lower bound from 7333940638915727360 to 7333940638742867968. Current Snapshot: MvccSnapshot[applied={T|T < 7333940638915727360}]
I20260927 12:40:51.324450 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:51.320824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54588 (local address 127.0.98.2:40853)
0927 12:40:51.320960 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:51.320965 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:51.320980 (+ 15us) server_negotiation.cc:407] Connection header received
0927 12:40:51.321052 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:51.321055 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:51.321103 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:51.321178 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:40:51.321448 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:51.322234 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:51.323486 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:51.323714 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:51.323796 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:51.324069 (+ 273us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:40:51.324135 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:40:51.324182 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:51.324294 (+ 112us) server_negotiation.cc:299] Negotiation successful
0927 12:40:51.324340 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 12:40:51.335207 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:51.331601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41645 (local address 127.0.98.2:40853)
0927 12:40:51.331714 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:51.331719 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:51.331968 (+ 249us) server_negotiation.cc:407] Connection header received
0927 12:40:51.332082 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:51.332084 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:51.332125 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:51.332185 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:40:51.332477 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:51.332942 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:51.334193 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:51.334400 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:51.334480 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:51.334792 (+ 312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:40:51.334865 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:40:51.334917 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:51.335030 (+ 113us) server_negotiation.cc:299] Negotiation successful
0927 12:40:51.335071 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":39}
I20260927 12:40:51.335351 2664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:51.331472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40853 (local address 127.0.98.2:41645)
0927 12:40:51.331854 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0927 12:40:51.331867 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:40:51.331995 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:40:51.332213 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:40:51.332217 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:40:51.332224 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 12:40:51.332378 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:51.332384 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:51.333037 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:51.333040 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:51.334078 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:51.334088 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:51.334503 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:51.334507 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:51.334605 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:51.334651 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 12:40:51.334944 (+ 293us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 12:40:51.334950 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 12:40:51.335015 (+ 65us) client_negotiation.cc:240] Negotiation successful
0927 12:40:51.335092 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"mutex_wait_us":17,"thread_start_us":140,"threads_started":1}
I20260927 12:40:51.341825 2559 raft_consensus.cc:1261] T 297b382dc1144ef791adad96d115cacf P 10b5013c5e0a4050ad9447ea19df245c [term 1 FOLLOWER]: Refusing update from remote peer 6092fc28ff124c22aef46193200cd07c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:40:51.341890 2279 raft_consensus.cc:1261] T 297b382dc1144ef791adad96d115cacf P af8e109d5d504a08aea237603b8b23a4 [term 1 FOLLOWER]: Refusing update from remote peer 6092fc28ff124c22aef46193200cd07c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:40:51.342193 2638 consensus_queue.cc:1035] T 297b382dc1144ef791adad96d115cacf P 6092fc28ff124c22aef46193200cd07c [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b5013c5e0a4050ad9447ea19df245c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:40:51.342329 2634 consensus_queue.cc:1035] T 297b382dc1144ef791adad96d115cacf P 6092fc28ff124c22aef46193200cd07c [LEADER]: Connected to new peer: Peer: permanent_uuid: "af8e109d5d504a08aea237603b8b23a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:40:51.388499 2121 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34790:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:40:51.388718 2121 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34790:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:40:51.391502 2121 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3e9ae20299c2487da31d96a749ba1a33: Sending DeleteTablet for 3 replicas of tablet 297b382dc1144ef791adad96d115cacf
I20260927 12:40:51.392198 2259 tablet_service.cc:1514] Processing DeleteTablet for tablet 297b382dc1144ef791adad96d115cacf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:40:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42600
I20260927 12:40:51.392268 2539 tablet_service.cc:1514] Processing DeleteTablet for tablet 297b382dc1144ef791adad96d115cacf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:40:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47786
I20260927 12:40:51.392820 2669 tablet_replica.cc:330] T 297b382dc1144ef791adad96d115cacf P af8e109d5d504a08aea237603b8b23a4: stopping tablet replica
I20260927 12:40:51.393143 2669 raft_consensus.cc:2229] T 297b382dc1144ef791adad96d115cacf P af8e109d5d504a08aea237603b8b23a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:40:51.393106 2398 tablet_service.cc:1514] Processing DeleteTablet for tablet 297b382dc1144ef791adad96d115cacf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:40:51 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54582
I20260927 12:40:51.393416 2669 raft_consensus.cc:2258] T 297b382dc1144ef791adad96d115cacf P af8e109d5d504a08aea237603b8b23a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:40:51.394527 2122 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34790:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 12:40:51.394961 2122 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:40:51.395282 2669 ts_tablet_manager.cc:1905] T 297b382dc1144ef791adad96d115cacf P af8e109d5d504a08aea237603b8b23a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:40:51.395931 2670 tablet_replica.cc:330] T 297b382dc1144ef791adad96d115cacf P 10b5013c5e0a4050ad9447ea19df245c: stopping tablet replica
I20260927 12:40:51.396224 2670 raft_consensus.cc:2229] T 297b382dc1144ef791adad96d115cacf P 10b5013c5e0a4050ad9447ea19df245c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:40:51.396456 2670 raft_consensus.cc:2258] T 297b382dc1144ef791adad96d115cacf P 10b5013c5e0a4050ad9447ea19df245c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:40:51.399590 2669 ts_tablet_manager.cc:1918] T 297b382dc1144ef791adad96d115cacf P af8e109d5d504a08aea237603b8b23a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 12:40:51.399744 2669 log.cc:1196] T 297b382dc1144ef791adad96d115cacf P af8e109d5d504a08aea237603b8b23a4: Deleting WAL directory at /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/minicluster-data/ts-0/wal/wals/297b382dc1144ef791adad96d115cacf
I20260927 12:40:51.400116 2669 ts_tablet_manager.cc:1939] T 297b382dc1144ef791adad96d115cacf P af8e109d5d504a08aea237603b8b23a4: Deleting consensus metadata
I20260927 12:40:51.401417 2109 catalog_manager.cc:4841] TS af8e109d5d504a08aea237603b8b23a4 (127.0.98.1:46773): tablet 297b382dc1144ef791adad96d115cacf (table test-table [id=2835283a5c7946f4b2fc127a97f44a49]) successfully deleted
I20260927 12:40:51.401504 2259 tablet_service.cc:1467] Processing CreateTablet for tablet 15f3cb9c58c74b4bae5c089685cb400a (DEFAULT_TABLE table=test-table [id=2a8cf723f56e469c8d8f0d51a41aa73c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:40:51.401783 2259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15f3cb9c58c74b4bae5c089685cb400a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:40:51.403985 2628 tablet_bootstrap.cc:492] T 15f3cb9c58c74b4bae5c089685cb400a P af8e109d5d504a08aea237603b8b23a4: Bootstrap starting.
I20260927 12:40:51.404171 2398 tablet_service.cc:1467] Processing CreateTablet for tablet 15f3cb9c58c74b4bae5c089685cb400a (DEFAULT_TABLE table=test-table [id=2a8cf723f56e469c8d8f0d51a41aa73c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:40:51.404451 2398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15f3cb9c58c74b4bae5c089685cb400a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:40:51.404757 2628 tablet_bootstrap.cc:654] T 15f3cb9c58c74b4bae5c089685cb400a P af8e109d5d504a08aea237603b8b23a4: Neither blocks nor log segments found. Creating new log.
I20260927 12:40:51.405882 2628 tablet_bootstrap.cc:492] T 15f3cb9c58c74b4bae5c089685cb400a P af8e109d5d504a08aea237603b8b23a4: No bootstrap required, opened a new log
I20260927 12:40:51.405958 2628 ts_tablet_manager.cc:1397] T 15f3cb9c58c74b4bae5c089685cb400a P af8e109d5d504a08aea237603b8b23a4: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260927 12:40:51.406330 2628 raft_consensus.cc:348] T 15f3cb9c58c74b4bae5c089685cb400a P af8e109d5d504a08aea237603b8b23a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8e109d5d504a08aea237603b8b23a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46773 } } peers { permanent_uuid: "6092fc28ff124c22aef46193200cd07c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40853 } } peers { permanent_uuid: "10b5013c5e0a4050ad9447ea19df245c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33869 } }
I20260927 12:40:51.406421 2628 raft_consensus.cc:374] T 15f3cb9c58c74b4bae5c089685cb400a P af8e109d5d504a08aea237603b8b23a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:40:51.406450 2628 raft_consensus.cc:729] T 15f3cb9c58c74b4bae5c089685cb400a P af8e109d5d504a08aea237603b8b23a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af8e109d5d504a08aea237603b8b23a4, State: Initialized, Role: FOLLOWER
I20260927 12:40:51.406554 2628 consensus_queue.cc:260] T 15f3cb9c58c74b4bae5c089685cb400a P af8e109d5d504a08aea237603b8b23a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8e109d5d504a08aea237603b8b23a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46773 } } peers { permanent_uuid: "6092fc28ff124c22aef46193200cd07c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40853 } } peers { permanent_uuid: "10b5013c5e0a4050ad9447ea19df245c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33869 } }
I20260927 12:40:51.406795 2628 ts_tablet_manager.cc:1428] T 15f3cb9c58c74b4bae5c089685cb400a P af8e109d5d504a08aea237603b8b23a4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 12:40:51.406947 2630 tablet_bootstrap.cc:492] T 15f3cb9c58c74b4bae5c089685cb400a P 6092fc28ff124c22aef46193200cd07c: Bootstrap starting.
I20260927 12:40:51.407882 2630 tablet_bootstrap.cc:654] T 15f3cb9c58c74b4bae5c089685cb400a P 6092fc28ff124c22aef46193200cd07c: Neither blocks nor log segments found. Creating new log.
I20260927 12:40:51.408118 2539 tablet_service.cc:1467] Processing CreateTablet for tablet 15f3cb9c58c74b4bae5c089685cb400a (DEFAULT_TABLE table=test-table [id=2a8cf723f56e469c8d8f0d51a41aa73c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:40:51.408394 2539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15f3cb9c58c74b4bae5c089685cb400a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:40:51.408849 2671 tablet_replica.cc:330] T 297b382dc1144ef791adad96d115cacf P 6092fc28ff124c22aef46193200cd07c: stopping tablet replica
I20260927 12:40:51.409199 2671 raft_consensus.cc:2229] T 297b382dc1144ef791adad96d115cacf P 6092fc28ff124c22aef46193200cd07c [term 1 LEADER]: Raft consensus shutting down.
I20260927 12:40:51.409654 2671 raft_consensus.cc:2258] T 297b382dc1144ef791adad96d115cacf P 6092fc28ff124c22aef46193200cd07c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:40:51.409837 2630 tablet_bootstrap.cc:492] T 15f3cb9c58c74b4bae5c089685cb400a P 6092fc28ff124c22aef46193200cd07c: No bootstrap required, opened a new log
I20260927 12:40:51.409926 2630 ts_tablet_manager.cc:1397] T 15f3cb9c58c74b4bae5c089685cb400a P 6092fc28ff124c22aef46193200cd07c: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 12:40:51.410378 2630 raft_consensus.cc:348] T 15f3cb9c58c74b4bae5c089685cb400a P 6092fc28ff124c22aef46193200cd07c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8e109d5d504a08aea237603b8b23a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46773 } } peers { permanent_uuid: "6092fc28ff124c22aef46193200cd07c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40853 } } peers { permanent_uuid: "10b5013c5e0a4050ad9447ea19df245c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33869 } }
I20260927 12:40:51.410497 2630 raft_consensus.cc:374] T 15f3cb9c58c74b4bae5c089685cb400a P 6092fc28ff124c22aef46193200cd07c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:40:51.410537 2630 raft_consensus.cc:729] T 15f3cb9c58c74b4bae5c089685cb400a P 6092fc28ff124c22aef46193200cd07c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6092fc28ff124c22aef46193200cd07c, State: Initialized, Role: FOLLOWER
I20260927 12:40:51.410670 2630 consensus_queue.cc:260] T 15f3cb9c58c74b4bae5c089685cb400a P 6092fc28ff124c22aef46193200cd07c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8e109d5d504a08aea237603b8b23a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46773 } } peers { permanent_uuid: "6092fc28ff124c22aef46193200cd07c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40853 } } peers { permanent_uuid: "10b5013c5e0a4050ad9447ea19df245c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33869 } }
I20260927 12:40:51.410774 2629 tablet_bootstrap.cc:492] T 15f3cb9c58c74b4bae5c089685cb400a P 10b5013c5e0a4050ad9447ea19df245c: Bootstrap starting.
I20260927 12:40:51.410919 2630 ts_tablet_manager.cc:1428] T 15f3cb9c58c74b4bae5c089685cb400a P 6092fc28ff124c22aef46193200cd07c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 12:40:51.411666 2629 tablet_bootstrap.cc:654] T 15f3cb9c58c74b4bae5c089685cb400a P 10b5013c5e0a4050ad9447ea19df245c: Neither blocks nor log segments found. Creating new log.
I20260927 12:40:51.412731 2670 ts_tablet_manager.cc:1905] T 297b382dc1144ef791adad96d115cacf P 10b5013c5e0a4050ad9447ea19df245c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:40:51.415505 2671 ts_tablet_manager.cc:1905] T 297b382dc1144ef791adad96d115cacf P 6092fc28ff124c22aef46193200cd07c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:40:51.418047 2671 ts_tablet_manager.cc:1918] T 297b382dc1144ef791adad96d115cacf P 6092fc28ff124c22aef46193200cd07c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 12:40:51.418143 2671 log.cc:1196] T 297b382dc1144ef791adad96d115cacf P 6092fc28ff124c22aef46193200cd07c: Deleting WAL directory at /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/minicluster-data/ts-1/wal/wals/297b382dc1144ef791adad96d115cacf
I20260927 12:40:51.418279 2629 tablet_bootstrap.cc:492] T 15f3cb9c58c74b4bae5c089685cb400a P 10b5013c5e0a4050ad9447ea19df245c: No bootstrap required, opened a new log
I20260927 12:40:51.418371 2629 ts_tablet_manager.cc:1397] T 15f3cb9c58c74b4bae5c089685cb400a P 10b5013c5e0a4050ad9447ea19df245c: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.001s
I20260927 12:40:51.418469 2671 ts_tablet_manager.cc:1939] T 297b382dc1144ef791adad96d115cacf P 6092fc28ff124c22aef46193200cd07c: Deleting consensus metadata
I20260927 12:40:51.418870 2629 raft_consensus.cc:348] T 15f3cb9c58c74b4bae5c089685cb400a P 10b5013c5e0a4050ad9447ea19df245c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8e109d5d504a08aea237603b8b23a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46773 } } peers { permanent_uuid: "6092fc28ff124c22aef46193200cd07c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40853 } } peers { permanent_uuid: "10b5013c5e0a4050ad9447ea19df245c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33869 } }
I20260927 12:40:51.418984 2629 raft_consensus.cc:374] T 15f3cb9c58c74b4bae5c089685cb400a P 10b5013c5e0a4050ad9447ea19df245c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:40:51.419023 2629 raft_consensus.cc:729] T 15f3cb9c58c74b4bae5c089685cb400a P 10b5013c5e0a4050ad9447ea19df245c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b5013c5e0a4050ad9447ea19df245c, State: Initialized, Role: FOLLOWER
I20260927 12:40:51.419149 2629 consensus_queue.cc:260] T 15f3cb9c58c74b4bae5c089685cb400a P 10b5013c5e0a4050ad9447ea19df245c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8e109d5d504a08aea237603b8b23a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46773 } } peers { permanent_uuid: "6092fc28ff124c22aef46193200cd07c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40853 } } peers { permanent_uuid: "10b5013c5e0a4050ad9447ea19df245c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33869 } }
I20260927 12:40:51.419431 2629 ts_tablet_manager.cc:1428] T 15f3cb9c58c74b4bae5c089685cb400a P 10b5013c5e0a4050ad9447ea19df245c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 12:40:51.419553 2109 catalog_manager.cc:4841] TS 6092fc28ff124c22aef46193200cd07c (127.0.98.2:40853): tablet 297b382dc1144ef791adad96d115cacf (table test-table [id=2835283a5c7946f4b2fc127a97f44a49]) successfully deleted
I20260927 12:40:51.420853 2670 ts_tablet_manager.cc:1918] T 297b382dc1144ef791adad96d115cacf P 10b5013c5e0a4050ad9447ea19df245c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 12:40:51.420950 2670 log.cc:1196] T 297b382dc1144ef791adad96d115cacf P 10b5013c5e0a4050ad9447ea19df245c: Deleting WAL directory at /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/minicluster-data/ts-2/wal/wals/297b382dc1144ef791adad96d115cacf
I20260927 12:40:51.421294 2670 ts_tablet_manager.cc:1939] T 297b382dc1144ef791adad96d115cacf P 10b5013c5e0a4050ad9447ea19df245c: Deleting consensus metadata
I20260927 12:40:51.422400 2107 catalog_manager.cc:4841] TS 10b5013c5e0a4050ad9447ea19df245c (127.0.98.3:33869): tablet 297b382dc1144ef791adad96d115cacf (table test-table [id=2835283a5c7946f4b2fc127a97f44a49]) successfully deleted
I20260927 12:40:51.462292 2638 raft_consensus.cc:482] T 15f3cb9c58c74b4bae5c089685cb400a P 6092fc28ff124c22aef46193200cd07c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:40:51.462478 2638 raft_consensus.cc:504] T 15f3cb9c58c74b4bae5c089685cb400a P 6092fc28ff124c22aef46193200cd07c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8e109d5d504a08aea237603b8b23a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46773 } } peers { permanent_uuid: "6092fc28ff124c22aef46193200cd07c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40853 } } peers { permanent_uuid: "10b5013c5e0a4050ad9447ea19df245c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33869 } }
I20260927 12:40:51.462911 2638 leader_election.cc:288] T 15f3cb9c58c74b4bae5c089685cb400a P 6092fc28ff124c22aef46193200cd07c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af8e109d5d504a08aea237603b8b23a4 (127.0.98.1:46773), 10b5013c5e0a4050ad9447ea19df245c (127.0.98.3:33869)
I20260927 12:40:51.463376 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15f3cb9c58c74b4bae5c089685cb400a" candidate_uuid: "6092fc28ff124c22aef46193200cd07c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af8e109d5d504a08aea237603b8b23a4" is_pre_election: true
I20260927 12:40:51.463392 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15f3cb9c58c74b4bae5c089685cb400a" candidate_uuid: "6092fc28ff124c22aef46193200cd07c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b5013c5e0a4050ad9447ea19df245c" is_pre_election: true
I20260927 12:40:51.463524 2279 raft_consensus.cc:2454] T 15f3cb9c58c74b4bae5c089685cb400a P af8e109d5d504a08aea237603b8b23a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6092fc28ff124c22aef46193200cd07c in term 0.
I20260927 12:40:51.463572 2559 raft_consensus.cc:2454] T 15f3cb9c58c74b4bae5c089685cb400a P 10b5013c5e0a4050ad9447ea19df245c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6092fc28ff124c22aef46193200cd07c in term 0.
I20260927 12:40:51.463881 2353 leader_election.cc:302] T 15f3cb9c58c74b4bae5c089685cb400a P 6092fc28ff124c22aef46193200cd07c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6092fc28ff124c22aef46193200cd07c, af8e109d5d504a08aea237603b8b23a4; no voters:
I20260927 12:40:51.464071 2638 raft_consensus.cc:2781] T 15f3cb9c58c74b4bae5c089685cb400a P 6092fc28ff124c22aef46193200cd07c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:40:51.464161 2638 raft_consensus.cc:482] T 15f3cb9c58c74b4bae5c089685cb400a P 6092fc28ff124c22aef46193200cd07c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:40:51.464217 2638 raft_consensus.cc:3037] T 15f3cb9c58c74b4bae5c089685cb400a P 6092fc28ff124c22aef46193200cd07c [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:40:51.465209 2638 raft_consensus.cc:504] T 15f3cb9c58c74b4bae5c089685cb400a P 6092fc28ff124c22aef46193200cd07c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8e109d5d504a08aea237603b8b23a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46773 } } peers { permanent_uuid: "6092fc28ff124c22aef46193200cd07c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40853 } } peers { permanent_uuid: "10b5013c5e0a4050ad9447ea19df245c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33869 } }
I20260927 12:40:51.465603 2638 leader_election.cc:288] T 15f3cb9c58c74b4bae5c089685cb400a P 6092fc28ff124c22aef46193200cd07c [CANDIDATE]: Term 1 election: Requested vote from peers af8e109d5d504a08aea237603b8b23a4 (127.0.98.1:46773), 10b5013c5e0a4050ad9447ea19df245c (127.0.98.3:33869)
I20260927 12:40:51.465950 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15f3cb9c58c74b4bae5c089685cb400a" candidate_uuid: "6092fc28ff124c22aef46193200cd07c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b5013c5e0a4050ad9447ea19df245c"
I20260927 12:40:51.465963 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15f3cb9c58c74b4bae5c089685cb400a" candidate_uuid: "6092fc28ff124c22aef46193200cd07c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af8e109d5d504a08aea237603b8b23a4"
I20260927 12:40:51.466068 2559 raft_consensus.cc:3037] T 15f3cb9c58c74b4bae5c089685cb400a P 10b5013c5e0a4050ad9447ea19df245c [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:40:51.466094 2279 raft_consensus.cc:3037] T 15f3cb9c58c74b4bae5c089685cb400a P af8e109d5d504a08aea237603b8b23a4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:40:51.467128 2559 raft_consensus.cc:2454] T 15f3cb9c58c74b4bae5c089685cb400a P 10b5013c5e0a4050ad9447ea19df245c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6092fc28ff124c22aef46193200cd07c in term 1.
I20260927 12:40:51.467120 2279 raft_consensus.cc:2454] T 15f3cb9c58c74b4bae5c089685cb400a P af8e109d5d504a08aea237603b8b23a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6092fc28ff124c22aef46193200cd07c in term 1.
I20260927 12:40:51.467451 2353 leader_election.cc:302] T 15f3cb9c58c74b4bae5c089685cb400a P 6092fc28ff124c22aef46193200cd07c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6092fc28ff124c22aef46193200cd07c, af8e109d5d504a08aea237603b8b23a4; no voters:
I20260927 12:40:51.467603 2638 raft_consensus.cc:2781] T 15f3cb9c58c74b4bae5c089685cb400a P 6092fc28ff124c22aef46193200cd07c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:40:51.467671 2638 raft_consensus.cc:686] T 15f3cb9c58c74b4bae5c089685cb400a P 6092fc28ff124c22aef46193200cd07c [term 1 LEADER]: Becoming Leader. State: Replica: 6092fc28ff124c22aef46193200cd07c, State: Running, Role: LEADER
I20260927 12:40:51.467795 2638 consensus_queue.cc:237] T 15f3cb9c58c74b4bae5c089685cb400a P 6092fc28ff124c22aef46193200cd07c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8e109d5d504a08aea237603b8b23a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46773 } } peers { permanent_uuid: "6092fc28ff124c22aef46193200cd07c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40853 } } peers { permanent_uuid: "10b5013c5e0a4050ad9447ea19df245c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33869 } }
I20260927 12:40:51.475579 2122 catalog_manager.cc:5495] T 15f3cb9c58c74b4bae5c089685cb400a P 6092fc28ff124c22aef46193200cd07c reported cstate change: term changed from 0 to 1, leader changed from <none> to 6092fc28ff124c22aef46193200cd07c (127.0.98.2). New cstate: current_term: 1 leader_uuid: "6092fc28ff124c22aef46193200cd07c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8e109d5d504a08aea237603b8b23a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6092fc28ff124c22aef46193200cd07c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10b5013c5e0a4050ad9447ea19df245c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20260927 12:40:51.505185 2559 raft_consensus.cc:1261] T 15f3cb9c58c74b4bae5c089685cb400a P 10b5013c5e0a4050ad9447ea19df245c [term 1 FOLLOWER]: Refusing update from remote peer 6092fc28ff124c22aef46193200cd07c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:40:51.505357 2279 raft_consensus.cc:1261] T 15f3cb9c58c74b4bae5c089685cb400a P af8e109d5d504a08aea237603b8b23a4 [term 1 FOLLOWER]: Refusing update from remote peer 6092fc28ff124c22aef46193200cd07c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:40:51.505635 2638 consensus_queue.cc:1035] T 15f3cb9c58c74b4bae5c089685cb400a P 6092fc28ff124c22aef46193200cd07c [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b5013c5e0a4050ad9447ea19df245c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:40:51.505810 2665 consensus_queue.cc:1035] T 15f3cb9c58c74b4bae5c089685cb400a P 6092fc28ff124c22aef46193200cd07c [LEADER]: Connected to new peer: Peer: permanent_uuid: "af8e109d5d504a08aea237603b8b23a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 12:40:51.518958 2653 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 12:40:53.069628 2121 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34790:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:40:53.069795 2121 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34790:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:40:53.072027 2121 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3e9ae20299c2487da31d96a749ba1a33: Sending DeleteTablet for 3 replicas of tablet 15f3cb9c58c74b4bae5c089685cb400a
I20260927 12:40:53.072535 2259 tablet_service.cc:1514] Processing DeleteTablet for tablet 15f3cb9c58c74b4bae5c089685cb400a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:40:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42600
I20260927 12:40:53.072547 2398 tablet_service.cc:1514] Processing DeleteTablet for tablet 15f3cb9c58c74b4bae5c089685cb400a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:40:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54582
I20260927 12:40:53.072861 2699 tablet_replica.cc:330] T 15f3cb9c58c74b4bae5c089685cb400a P af8e109d5d504a08aea237603b8b23a4: stopping tablet replica
I20260927 12:40:53.072878 2700 tablet_replica.cc:330] T 15f3cb9c58c74b4bae5c089685cb400a P 6092fc28ff124c22aef46193200cd07c: stopping tablet replica
I20260927 12:40:53.072860 2539 tablet_service.cc:1514] Processing DeleteTablet for tablet 15f3cb9c58c74b4bae5c089685cb400a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:40:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47786
I20260927 12:40:53.073200 2700 raft_consensus.cc:2229] T 15f3cb9c58c74b4bae5c089685cb400a P 6092fc28ff124c22aef46193200cd07c [term 1 LEADER]: Raft consensus shutting down.
I20260927 12:40:53.073396 2700 raft_consensus.cc:2258] T 15f3cb9c58c74b4bae5c089685cb400a P 6092fc28ff124c22aef46193200cd07c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:40:53.073961 2701 tablet_replica.cc:330] T 15f3cb9c58c74b4bae5c089685cb400a P 10b5013c5e0a4050ad9447ea19df245c: stopping tablet replica
I20260927 12:40:53.074020 2700 ts_tablet_manager.cc:1905] T 15f3cb9c58c74b4bae5c089685cb400a P 6092fc28ff124c22aef46193200cd07c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:40:53.074105 2701 raft_consensus.cc:2229] T 15f3cb9c58c74b4bae5c089685cb400a P 10b5013c5e0a4050ad9447ea19df245c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:40:53.074229 2701 raft_consensus.cc:2258] T 15f3cb9c58c74b4bae5c089685cb400a P 10b5013c5e0a4050ad9447ea19df245c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:40:53.074816 2701 ts_tablet_manager.cc:1905] T 15f3cb9c58c74b4bae5c089685cb400a P 10b5013c5e0a4050ad9447ea19df245c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:40:53.075400 2699 raft_consensus.cc:2229] T 15f3cb9c58c74b4bae5c089685cb400a P af8e109d5d504a08aea237603b8b23a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:40:53.075555 2699 raft_consensus.cc:2258] T 15f3cb9c58c74b4bae5c089685cb400a P af8e109d5d504a08aea237603b8b23a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:40:53.076197 2699 ts_tablet_manager.cc:1905] T 15f3cb9c58c74b4bae5c089685cb400a P af8e109d5d504a08aea237603b8b23a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:40:53.077243 2701 ts_tablet_manager.cc:1918] T 15f3cb9c58c74b4bae5c089685cb400a P 10b5013c5e0a4050ad9447ea19df245c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 12:40:53.077314 2701 log.cc:1196] T 15f3cb9c58c74b4bae5c089685cb400a P 10b5013c5e0a4050ad9447ea19df245c: Deleting WAL directory at /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/minicluster-data/ts-2/wal/wals/15f3cb9c58c74b4bae5c089685cb400a
I20260927 12:40:53.077407 2700 ts_tablet_manager.cc:1918] T 15f3cb9c58c74b4bae5c089685cb400a P 6092fc28ff124c22aef46193200cd07c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 12:40:53.077459 2700 log.cc:1196] T 15f3cb9c58c74b4bae5c089685cb400a P 6092fc28ff124c22aef46193200cd07c: Deleting WAL directory at /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/minicluster-data/ts-1/wal/wals/15f3cb9c58c74b4bae5c089685cb400a
I20260927 12:40:53.077595 2701 ts_tablet_manager.cc:1939] T 15f3cb9c58c74b4bae5c089685cb400a P 10b5013c5e0a4050ad9447ea19df245c: Deleting consensus metadata
I20260927 12:40:53.077739 2700 ts_tablet_manager.cc:1939] T 15f3cb9c58c74b4bae5c089685cb400a P 6092fc28ff124c22aef46193200cd07c: Deleting consensus metadata
I20260927 12:40:53.078289 2107 catalog_manager.cc:4841] TS 10b5013c5e0a4050ad9447ea19df245c (127.0.98.3:33869): tablet 15f3cb9c58c74b4bae5c089685cb400a (table test-table [id=2a8cf723f56e469c8d8f0d51a41aa73c]) successfully deleted
I20260927 12:40:53.078366 2109 catalog_manager.cc:4841] TS 6092fc28ff124c22aef46193200cd07c (127.0.98.2:40853): tablet 15f3cb9c58c74b4bae5c089685cb400a (table test-table [id=2a8cf723f56e469c8d8f0d51a41aa73c]) successfully deleted
I20260927 12:40:53.079028 2699 ts_tablet_manager.cc:1918] T 15f3cb9c58c74b4bae5c089685cb400a P af8e109d5d504a08aea237603b8b23a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 12:40:53.079108 2699 log.cc:1196] T 15f3cb9c58c74b4bae5c089685cb400a P af8e109d5d504a08aea237603b8b23a4: Deleting WAL directory at /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/minicluster-data/ts-0/wal/wals/15f3cb9c58c74b4bae5c089685cb400a
I20260927 12:40:53.079378 2699 ts_tablet_manager.cc:1939] T 15f3cb9c58c74b4bae5c089685cb400a P af8e109d5d504a08aea237603b8b23a4: Deleting consensus metadata
I20260927 12:40:53.080117 2109 catalog_manager.cc:4841] TS af8e109d5d504a08aea237603b8b23a4 (127.0.98.1:46773): tablet 15f3cb9c58c74b4bae5c089685cb400a (table test-table [id=2a8cf723f56e469c8d8f0d51a41aa73c]) successfully deleted
Sep 27 12:40:53 dist-test-slave-bnc1 krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512853, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 12:40:53 dist-test-slave-bnc1 krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512853, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:40:53.101255 2709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:53.092609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53742 (local address 127.0.98.62:45763)
0927 12:40:53.092912 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:53.092919 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:53.092936 (+ 17us) server_negotiation.cc:407] Connection header received
0927 12:40:53.092978 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:53.092980 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:53.093020 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:53.093098 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:40:53.093893 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:53.094469 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:53.095493 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:53.095749 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:53.095826 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:53.098324 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:40:53.098327 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:40:53.098329 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:40:53.098350 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:40:53.099845 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:53.100210 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:53.100212 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:53.100214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:53.100320 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:53.100566 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:53.100568 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:53.100569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:53.100731 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:40:53.100829 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:53.101033 (+ 204us) server_negotiation.cc:299] Negotiation successful
0927 12:40:53.101121 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":135,"threads_started":1}
I20260927 12:40:53.103734 2121 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53742:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 12:40:53.104203 2121 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:40:53.108887 2259 tablet_service.cc:1467] Processing CreateTablet for tablet cfe255b2a15547c4b98f43ebc154fa50 (DEFAULT_TABLE table=test-table [id=23f2e5bad43b49619205ef9e106b8c59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:40:53.108980 2398 tablet_service.cc:1467] Processing CreateTablet for tablet cfe255b2a15547c4b98f43ebc154fa50 (DEFAULT_TABLE table=test-table [id=23f2e5bad43b49619205ef9e106b8c59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:40:53.109037 2539 tablet_service.cc:1467] Processing CreateTablet for tablet cfe255b2a15547c4b98f43ebc154fa50 (DEFAULT_TABLE table=test-table [id=23f2e5bad43b49619205ef9e106b8c59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:40:53.109227 2259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfe255b2a15547c4b98f43ebc154fa50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:40:53.109290 2539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfe255b2a15547c4b98f43ebc154fa50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:40:53.109326 2398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfe255b2a15547c4b98f43ebc154fa50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:40:53.111384 2710 tablet_bootstrap.cc:492] T cfe255b2a15547c4b98f43ebc154fa50 P 6092fc28ff124c22aef46193200cd07c: Bootstrap starting.
I20260927 12:40:53.111883 2711 tablet_bootstrap.cc:492] T cfe255b2a15547c4b98f43ebc154fa50 P 10b5013c5e0a4050ad9447ea19df245c: Bootstrap starting.
I20260927 12:40:53.112363 2710 tablet_bootstrap.cc:654] T cfe255b2a15547c4b98f43ebc154fa50 P 6092fc28ff124c22aef46193200cd07c: Neither blocks nor log segments found. Creating new log.
I20260927 12:40:53.112901 2711 tablet_bootstrap.cc:654] T cfe255b2a15547c4b98f43ebc154fa50 P 10b5013c5e0a4050ad9447ea19df245c: Neither blocks nor log segments found. Creating new log.
I20260927 12:40:53.113440 2710 tablet_bootstrap.cc:492] T cfe255b2a15547c4b98f43ebc154fa50 P 6092fc28ff124c22aef46193200cd07c: No bootstrap required, opened a new log
I20260927 12:40:53.113471 2712 tablet_bootstrap.cc:492] T cfe255b2a15547c4b98f43ebc154fa50 P af8e109d5d504a08aea237603b8b23a4: Bootstrap starting.
I20260927 12:40:53.113533 2710 ts_tablet_manager.cc:1397] T cfe255b2a15547c4b98f43ebc154fa50 P 6092fc28ff124c22aef46193200cd07c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 12:40:53.113994 2710 raft_consensus.cc:348] T cfe255b2a15547c4b98f43ebc154fa50 P 6092fc28ff124c22aef46193200cd07c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8e109d5d504a08aea237603b8b23a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46773 } } peers { permanent_uuid: "6092fc28ff124c22aef46193200cd07c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40853 } } peers { permanent_uuid: "10b5013c5e0a4050ad9447ea19df245c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33869 } }
I20260927 12:40:53.114118 2710 raft_consensus.cc:374] T cfe255b2a15547c4b98f43ebc154fa50 P 6092fc28ff124c22aef46193200cd07c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:40:53.114159 2710 raft_consensus.cc:729] T cfe255b2a15547c4b98f43ebc154fa50 P 6092fc28ff124c22aef46193200cd07c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6092fc28ff124c22aef46193200cd07c, State: Initialized, Role: FOLLOWER
I20260927 12:40:53.114307 2710 consensus_queue.cc:260] T cfe255b2a15547c4b98f43ebc154fa50 P 6092fc28ff124c22aef46193200cd07c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8e109d5d504a08aea237603b8b23a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46773 } } peers { permanent_uuid: "6092fc28ff124c22aef46193200cd07c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40853 } } peers { permanent_uuid: "10b5013c5e0a4050ad9447ea19df245c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33869 } }
I20260927 12:40:53.114548 2711 tablet_bootstrap.cc:492] T cfe255b2a15547c4b98f43ebc154fa50 P 10b5013c5e0a4050ad9447ea19df245c: No bootstrap required, opened a new log
I20260927 12:40:53.114624 2710 ts_tablet_manager.cc:1428] T cfe255b2a15547c4b98f43ebc154fa50 P 6092fc28ff124c22aef46193200cd07c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 12:40:53.114715 2712 tablet_bootstrap.cc:654] T cfe255b2a15547c4b98f43ebc154fa50 P af8e109d5d504a08aea237603b8b23a4: Neither blocks nor log segments found. Creating new log.
I20260927 12:40:53.114717 2711 ts_tablet_manager.cc:1397] T cfe255b2a15547c4b98f43ebc154fa50 P 10b5013c5e0a4050ad9447ea19df245c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 12:40:53.115248 2711 raft_consensus.cc:348] T cfe255b2a15547c4b98f43ebc154fa50 P 10b5013c5e0a4050ad9447ea19df245c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8e109d5d504a08aea237603b8b23a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46773 } } peers { permanent_uuid: "6092fc28ff124c22aef46193200cd07c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40853 } } peers { permanent_uuid: "10b5013c5e0a4050ad9447ea19df245c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33869 } }
I20260927 12:40:53.115415 2711 raft_consensus.cc:374] T cfe255b2a15547c4b98f43ebc154fa50 P 10b5013c5e0a4050ad9447ea19df245c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:40:53.115461 2711 raft_consensus.cc:729] T cfe255b2a15547c4b98f43ebc154fa50 P 10b5013c5e0a4050ad9447ea19df245c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b5013c5e0a4050ad9447ea19df245c, State: Initialized, Role: FOLLOWER
I20260927 12:40:53.115593 2711 consensus_queue.cc:260] T cfe255b2a15547c4b98f43ebc154fa50 P 10b5013c5e0a4050ad9447ea19df245c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8e109d5d504a08aea237603b8b23a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46773 } } peers { permanent_uuid: "6092fc28ff124c22aef46193200cd07c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40853 } } peers { permanent_uuid: "10b5013c5e0a4050ad9447ea19df245c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33869 } }
I20260927 12:40:53.115988 2711 ts_tablet_manager.cc:1428] T cfe255b2a15547c4b98f43ebc154fa50 P 10b5013c5e0a4050ad9447ea19df245c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 12:40:53.116047 2712 tablet_bootstrap.cc:492] T cfe255b2a15547c4b98f43ebc154fa50 P af8e109d5d504a08aea237603b8b23a4: No bootstrap required, opened a new log
I20260927 12:40:53.116129 2712 ts_tablet_manager.cc:1397] T cfe255b2a15547c4b98f43ebc154fa50 P af8e109d5d504a08aea237603b8b23a4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 12:40:53.117000 2712 raft_consensus.cc:348] T cfe255b2a15547c4b98f43ebc154fa50 P af8e109d5d504a08aea237603b8b23a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8e109d5d504a08aea237603b8b23a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46773 } } peers { permanent_uuid: "6092fc28ff124c22aef46193200cd07c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40853 } } peers { permanent_uuid: "10b5013c5e0a4050ad9447ea19df245c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33869 } }
I20260927 12:40:53.117120 2712 raft_consensus.cc:374] T cfe255b2a15547c4b98f43ebc154fa50 P af8e109d5d504a08aea237603b8b23a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:40:53.117157 2712 raft_consensus.cc:729] T cfe255b2a15547c4b98f43ebc154fa50 P af8e109d5d504a08aea237603b8b23a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af8e109d5d504a08aea237603b8b23a4, State: Initialized, Role: FOLLOWER
I20260927 12:40:53.117269 2712 consensus_queue.cc:260] T cfe255b2a15547c4b98f43ebc154fa50 P af8e109d5d504a08aea237603b8b23a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8e109d5d504a08aea237603b8b23a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46773 } } peers { permanent_uuid: "6092fc28ff124c22aef46193200cd07c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40853 } } peers { permanent_uuid: "10b5013c5e0a4050ad9447ea19df245c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33869 } }
I20260927 12:40:53.117650 2712 ts_tablet_manager.cc:1428] T cfe255b2a15547c4b98f43ebc154fa50 P af8e109d5d504a08aea237603b8b23a4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 12:40:53.302160 2686 raft_consensus.cc:482] T cfe255b2a15547c4b98f43ebc154fa50 P 6092fc28ff124c22aef46193200cd07c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:40:53.302286 2686 raft_consensus.cc:504] T cfe255b2a15547c4b98f43ebc154fa50 P 6092fc28ff124c22aef46193200cd07c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8e109d5d504a08aea237603b8b23a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46773 } } peers { permanent_uuid: "6092fc28ff124c22aef46193200cd07c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40853 } } peers { permanent_uuid: "10b5013c5e0a4050ad9447ea19df245c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33869 } }
I20260927 12:40:53.302788 2686 leader_election.cc:288] T cfe255b2a15547c4b98f43ebc154fa50 P 6092fc28ff124c22aef46193200cd07c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af8e109d5d504a08aea237603b8b23a4 (127.0.98.1:46773), 10b5013c5e0a4050ad9447ea19df245c (127.0.98.3:33869)
I20260927 12:40:53.303164 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfe255b2a15547c4b98f43ebc154fa50" candidate_uuid: "6092fc28ff124c22aef46193200cd07c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af8e109d5d504a08aea237603b8b23a4" is_pre_election: true
I20260927 12:40:53.303215 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfe255b2a15547c4b98f43ebc154fa50" candidate_uuid: "6092fc28ff124c22aef46193200cd07c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b5013c5e0a4050ad9447ea19df245c" is_pre_election: true
I20260927 12:40:53.303359 2279 raft_consensus.cc:2454] T cfe255b2a15547c4b98f43ebc154fa50 P af8e109d5d504a08aea237603b8b23a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6092fc28ff124c22aef46193200cd07c in term 0.
I20260927 12:40:53.303375 2559 raft_consensus.cc:2454] T cfe255b2a15547c4b98f43ebc154fa50 P 10b5013c5e0a4050ad9447ea19df245c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6092fc28ff124c22aef46193200cd07c in term 0.
I20260927 12:40:53.303714 2353 leader_election.cc:302] T cfe255b2a15547c4b98f43ebc154fa50 P 6092fc28ff124c22aef46193200cd07c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6092fc28ff124c22aef46193200cd07c, af8e109d5d504a08aea237603b8b23a4; no voters:
I20260927 12:40:53.303889 2686 raft_consensus.cc:2781] T cfe255b2a15547c4b98f43ebc154fa50 P 6092fc28ff124c22aef46193200cd07c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:40:53.303953 2686 raft_consensus.cc:482] T cfe255b2a15547c4b98f43ebc154fa50 P 6092fc28ff124c22aef46193200cd07c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:40:53.304025 2686 raft_consensus.cc:3037] T cfe255b2a15547c4b98f43ebc154fa50 P 6092fc28ff124c22aef46193200cd07c [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:40:53.305027 2686 raft_consensus.cc:504] T cfe255b2a15547c4b98f43ebc154fa50 P 6092fc28ff124c22aef46193200cd07c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8e109d5d504a08aea237603b8b23a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46773 } } peers { permanent_uuid: "6092fc28ff124c22aef46193200cd07c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40853 } } peers { permanent_uuid: "10b5013c5e0a4050ad9447ea19df245c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33869 } }
I20260927 12:40:53.305389 2686 leader_election.cc:288] T cfe255b2a15547c4b98f43ebc154fa50 P 6092fc28ff124c22aef46193200cd07c [CANDIDATE]: Term 1 election: Requested vote from peers af8e109d5d504a08aea237603b8b23a4 (127.0.98.1:46773), 10b5013c5e0a4050ad9447ea19df245c (127.0.98.3:33869)
I20260927 12:40:53.305718 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfe255b2a15547c4b98f43ebc154fa50" candidate_uuid: "6092fc28ff124c22aef46193200cd07c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af8e109d5d504a08aea237603b8b23a4"
I20260927 12:40:53.305823 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfe255b2a15547c4b98f43ebc154fa50" candidate_uuid: "6092fc28ff124c22aef46193200cd07c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10b5013c5e0a4050ad9447ea19df245c"
I20260927 12:40:53.305864 2279 raft_consensus.cc:3037] T cfe255b2a15547c4b98f43ebc154fa50 P af8e109d5d504a08aea237603b8b23a4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:40:53.305943 2559 raft_consensus.cc:3037] T cfe255b2a15547c4b98f43ebc154fa50 P 10b5013c5e0a4050ad9447ea19df245c [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:40:53.306783 2559 raft_consensus.cc:2454] T cfe255b2a15547c4b98f43ebc154fa50 P 10b5013c5e0a4050ad9447ea19df245c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6092fc28ff124c22aef46193200cd07c in term 1.
I20260927 12:40:53.306788 2279 raft_consensus.cc:2454] T cfe255b2a15547c4b98f43ebc154fa50 P af8e109d5d504a08aea237603b8b23a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6092fc28ff124c22aef46193200cd07c in term 1.
I20260927 12:40:53.307065 2353 leader_election.cc:302] T cfe255b2a15547c4b98f43ebc154fa50 P 6092fc28ff124c22aef46193200cd07c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6092fc28ff124c22aef46193200cd07c, af8e109d5d504a08aea237603b8b23a4; no voters:
I20260927 12:40:53.307222 2686 raft_consensus.cc:2781] T cfe255b2a15547c4b98f43ebc154fa50 P 6092fc28ff124c22aef46193200cd07c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:40:53.307314 2686 raft_consensus.cc:686] T cfe255b2a15547c4b98f43ebc154fa50 P 6092fc28ff124c22aef46193200cd07c [term 1 LEADER]: Becoming Leader. State: Replica: 6092fc28ff124c22aef46193200cd07c, State: Running, Role: LEADER
I20260927 12:40:53.307489 2686 consensus_queue.cc:237] T cfe255b2a15547c4b98f43ebc154fa50 P 6092fc28ff124c22aef46193200cd07c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8e109d5d504a08aea237603b8b23a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46773 } } peers { permanent_uuid: "6092fc28ff124c22aef46193200cd07c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40853 } } peers { permanent_uuid: "10b5013c5e0a4050ad9447ea19df245c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33869 } }
I20260927 12:40:53.309130 2121 catalog_manager.cc:5495] T cfe255b2a15547c4b98f43ebc154fa50 P 6092fc28ff124c22aef46193200cd07c reported cstate change: term changed from 0 to 1, leader changed from <none> to 6092fc28ff124c22aef46193200cd07c (127.0.98.2). New cstate: current_term: 1 leader_uuid: "6092fc28ff124c22aef46193200cd07c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8e109d5d504a08aea237603b8b23a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6092fc28ff124c22aef46193200cd07c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10b5013c5e0a4050ad9447ea19df245c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33869 } health_report { overall_health: UNKNOWN } } }
I20260927 12:40:53.357985 2720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:53.354677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57280 (local address 127.0.98.2:40853)
0927 12:40:53.354946 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:53.354955 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:53.354967 (+ 12us) server_negotiation.cc:407] Connection header received
0927 12:40:53.355012 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:53.355014 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:53.355052 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:53.355140 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:40:53.355393 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:53.355933 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:53.356880 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:53.357126 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:53.357236 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:53.357505 (+ 269us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:40:53.357602 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:40:53.357683 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:53.357779 (+ 96us) server_negotiation.cc:299] Negotiation successful
0927 12:40:53.357830 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":94,"threads_started":1}
I20260927 12:40:53.363803 2279 raft_consensus.cc:1261] T cfe255b2a15547c4b98f43ebc154fa50 P af8e109d5d504a08aea237603b8b23a4 [term 1 FOLLOWER]: Refusing update from remote peer 6092fc28ff124c22aef46193200cd07c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:40:53.363865 2559 raft_consensus.cc:1261] T cfe255b2a15547c4b98f43ebc154fa50 P 10b5013c5e0a4050ad9447ea19df245c [term 1 FOLLOWER]: Refusing update from remote peer 6092fc28ff124c22aef46193200cd07c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:40:53.364197 2686 consensus_queue.cc:1035] T cfe255b2a15547c4b98f43ebc154fa50 P 6092fc28ff124c22aef46193200cd07c [LEADER]: Connected to new peer: Peer: permanent_uuid: "10b5013c5e0a4050ad9447ea19df245c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:40:53.364336 2638 consensus_queue.cc:1035] T cfe255b2a15547c4b98f43ebc154fa50 P 6092fc28ff124c22aef46193200cd07c [LEADER]: Connected to new peer: Peer: permanent_uuid: "af8e109d5d504a08aea237603b8b23a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:40:53.393603 2122 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:40:53.393781 2122 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:40:53.395829 2122 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3e9ae20299c2487da31d96a749ba1a33: Sending DeleteTablet for 3 replicas of tablet cfe255b2a15547c4b98f43ebc154fa50
I20260927 12:40:53.396299 2259 tablet_service.cc:1514] Processing DeleteTablet for tablet cfe255b2a15547c4b98f43ebc154fa50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:40:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42600
I20260927 12:40:53.396365 2398 tablet_service.cc:1514] Processing DeleteTablet for tablet cfe255b2a15547c4b98f43ebc154fa50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:40:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54582
I20260927 12:40:53.396467 2699 tablet_replica.cc:330] T cfe255b2a15547c4b98f43ebc154fa50 P af8e109d5d504a08aea237603b8b23a4: stopping tablet replica
I20260927 12:40:53.396580 2699 raft_consensus.cc:2229] T cfe255b2a15547c4b98f43ebc154fa50 P af8e109d5d504a08aea237603b8b23a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:40:53.396700 2699 raft_consensus.cc:2258] T cfe255b2a15547c4b98f43ebc154fa50 P af8e109d5d504a08aea237603b8b23a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:40:53.396728 2539 tablet_service.cc:1514] Processing DeleteTablet for tablet cfe255b2a15547c4b98f43ebc154fa50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:40:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47786
I20260927 12:40:53.396855 2700 tablet_replica.cc:330] T cfe255b2a15547c4b98f43ebc154fa50 P 6092fc28ff124c22aef46193200cd07c: stopping tablet replica
I20260927 12:40:53.396948 2701 tablet_replica.cc:330] T cfe255b2a15547c4b98f43ebc154fa50 P 10b5013c5e0a4050ad9447ea19df245c: stopping tablet replica
I20260927 12:40:53.396958 2700 raft_consensus.cc:2229] T cfe255b2a15547c4b98f43ebc154fa50 P 6092fc28ff124c22aef46193200cd07c [term 1 LEADER]: Raft consensus shutting down.
I20260927 12:40:53.397032 2701 raft_consensus.cc:2229] T cfe255b2a15547c4b98f43ebc154fa50 P 10b5013c5e0a4050ad9447ea19df245c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:40:53.397107 2701 raft_consensus.cc:2258] T cfe255b2a15547c4b98f43ebc154fa50 P 10b5013c5e0a4050ad9447ea19df245c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:40:53.397106 2700 raft_consensus.cc:2258] T cfe255b2a15547c4b98f43ebc154fa50 P 6092fc28ff124c22aef46193200cd07c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:40:53.397857 2700 ts_tablet_manager.cc:1905] T cfe255b2a15547c4b98f43ebc154fa50 P 6092fc28ff124c22aef46193200cd07c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:40:53.397886 2699 ts_tablet_manager.cc:1905] T cfe255b2a15547c4b98f43ebc154fa50 P af8e109d5d504a08aea237603b8b23a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:40:53.398211 2122 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53742:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 12:40:53.398699 2122 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:40:53.400266 2700 ts_tablet_manager.cc:1918] T cfe255b2a15547c4b98f43ebc154fa50 P 6092fc28ff124c22aef46193200cd07c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 12:40:53.400300 2699 ts_tablet_manager.cc:1918] T cfe255b2a15547c4b98f43ebc154fa50 P af8e109d5d504a08aea237603b8b23a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 12:40:53.400362 2699 log.cc:1196] T cfe255b2a15547c4b98f43ebc154fa50 P af8e109d5d504a08aea237603b8b23a4: Deleting WAL directory at /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/minicluster-data/ts-0/wal/wals/cfe255b2a15547c4b98f43ebc154fa50
I20260927 12:40:53.400357 2700 log.cc:1196] T cfe255b2a15547c4b98f43ebc154fa50 P 6092fc28ff124c22aef46193200cd07c: Deleting WAL directory at /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/minicluster-data/ts-1/wal/wals/cfe255b2a15547c4b98f43ebc154fa50
I20260927 12:40:53.400604 2699 ts_tablet_manager.cc:1939] T cfe255b2a15547c4b98f43ebc154fa50 P af8e109d5d504a08aea237603b8b23a4: Deleting consensus metadata
I20260927 12:40:53.400604 2700 ts_tablet_manager.cc:1939] T cfe255b2a15547c4b98f43ebc154fa50 P 6092fc28ff124c22aef46193200cd07c: Deleting consensus metadata
I20260927 12:40:53.401288 2109 catalog_manager.cc:4841] TS af8e109d5d504a08aea237603b8b23a4 (127.0.98.1:46773): tablet cfe255b2a15547c4b98f43ebc154fa50 (table test-table [id=23f2e5bad43b49619205ef9e106b8c59]) successfully deleted
I20260927 12:40:53.402037 2109 catalog_manager.cc:4841] TS 6092fc28ff124c22aef46193200cd07c (127.0.98.2:40853): tablet cfe255b2a15547c4b98f43ebc154fa50 (table test-table [id=23f2e5bad43b49619205ef9e106b8c59]) successfully deleted
I20260927 12:40:53.404585 2259 tablet_service.cc:1467] Processing CreateTablet for tablet fe6108b3dfcd43d7b6063c49be1f9641 (DEFAULT_TABLE table=test-table [id=c24af1bdc2534bd9a43e81da7292804b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:40:53.404610 2398 tablet_service.cc:1467] Processing CreateTablet for tablet fe6108b3dfcd43d7b6063c49be1f9641 (DEFAULT_TABLE table=test-table [id=c24af1bdc2534bd9a43e81da7292804b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:40:53.404820 2259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe6108b3dfcd43d7b6063c49be1f9641. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:40:53.404843 2398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe6108b3dfcd43d7b6063c49be1f9641. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:40:53.406124 2701 ts_tablet_manager.cc:1905] T cfe255b2a15547c4b98f43ebc154fa50 P 10b5013c5e0a4050ad9447ea19df245c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:40:53.406725 2712 tablet_bootstrap.cc:492] T fe6108b3dfcd43d7b6063c49be1f9641 P af8e109d5d504a08aea237603b8b23a4: Bootstrap starting.
I20260927 12:40:53.406867 2710 tablet_bootstrap.cc:492] T fe6108b3dfcd43d7b6063c49be1f9641 P 6092fc28ff124c22aef46193200cd07c: Bootstrap starting.
I20260927 12:40:53.407070 2539 tablet_service.cc:1467] Processing CreateTablet for tablet fe6108b3dfcd43d7b6063c49be1f9641 (DEFAULT_TABLE table=test-table [id=c24af1bdc2534bd9a43e81da7292804b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:40:53.407315 2539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe6108b3dfcd43d7b6063c49be1f9641. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:40:53.407655 2712 tablet_bootstrap.cc:654] T fe6108b3dfcd43d7b6063c49be1f9641 P af8e109d5d504a08aea237603b8b23a4: Neither blocks nor log segments found. Creating new log.
I20260927 12:40:53.407706 2710 tablet_bootstrap.cc:654] T fe6108b3dfcd43d7b6063c49be1f9641 P 6092fc28ff124c22aef46193200cd07c: Neither blocks nor log segments found. Creating new log.
I20260927 12:40:53.408607 2701 ts_tablet_manager.cc:1918] T cfe255b2a15547c4b98f43ebc154fa50 P 10b5013c5e0a4050ad9447ea19df245c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 12:40:53.408694 2701 log.cc:1196] T cfe255b2a15547c4b98f43ebc154fa50 P 10b5013c5e0a4050ad9447ea19df245c: Deleting WAL directory at /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/minicluster-data/ts-2/wal/wals/cfe255b2a15547c4b98f43ebc154fa50
I20260927 12:40:53.408944 2701 ts_tablet_manager.cc:1939] T cfe255b2a15547c4b98f43ebc154fa50 P 10b5013c5e0a4050ad9447ea19df245c: Deleting consensus metadata
I20260927 12:40:53.409405 2711 tablet_bootstrap.cc:492] T fe6108b3dfcd43d7b6063c49be1f9641 P 10b5013c5e0a4050ad9447ea19df245c: Bootstrap starting.
I20260927 12:40:53.409651 2107 catalog_manager.cc:4841] TS 10b5013c5e0a4050ad9447ea19df245c (127.0.98.3:33869): tablet cfe255b2a15547c4b98f43ebc154fa50 (table test-table [id=23f2e5bad43b49619205ef9e106b8c59]) successfully deleted
I20260927 12:40:53.410250 2711 tablet_bootstrap.cc:654] T fe6108b3dfcd43d7b6063c49be1f9641 P 10b5013c5e0a4050ad9447ea19df245c: Neither blocks nor log segments found. Creating new log.
I20260927 12:40:53.410439 2712 tablet_bootstrap.cc:492] T fe6108b3dfcd43d7b6063c49be1f9641 P af8e109d5d504a08aea237603b8b23a4: No bootstrap required, opened a new log
I20260927 12:40:53.410517 2712 ts_tablet_manager.cc:1397] T fe6108b3dfcd43d7b6063c49be1f9641 P af8e109d5d504a08aea237603b8b23a4: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260927 12:40:53.410724 2710 tablet_bootstrap.cc:492] T fe6108b3dfcd43d7b6063c49be1f9641 P 6092fc28ff124c22aef46193200cd07c: No bootstrap required, opened a new log
I20260927 12:40:53.410807 2710 ts_tablet_manager.cc:1397] T fe6108b3dfcd43d7b6063c49be1f9641 P 6092fc28ff124c22aef46193200cd07c: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260927 12:40:53.410939 2712 raft_consensus.cc:348] T fe6108b3dfcd43d7b6063c49be1f9641 P af8e109d5d504a08aea237603b8b23a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8e109d5d504a08aea237603b8b23a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46773 } } peers { permanent_uuid: "6092fc28ff124c22aef46193200cd07c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40853 } } peers { permanent_uuid: "10b5013c5e0a4050ad9447ea19df245c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33869 } }
I20260927 12:40:53.411055 2712 raft_consensus.cc:374] T fe6108b3dfcd43d7b6063c49be1f9641 P af8e109d5d504a08aea237603b8b23a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:40:53.411090 2712 raft_consensus.cc:729] T fe6108b3dfcd43d7b6063c49be1f9641 P af8e109d5d504a08aea237603b8b23a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af8e109d5d504a08aea237603b8b23a4, State: Initialized, Role: FOLLOWER
I20260927 12:40:53.411175 2710 raft_consensus.cc:348] T fe6108b3dfcd43d7b6063c49be1f9641 P 6092fc28ff124c22aef46193200cd07c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8e109d5d504a08aea237603b8b23a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46773 } } peers { permanent_uuid: "6092fc28ff124c22aef46193200cd07c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40853 } } peers { permanent_uuid: "10b5013c5e0a4050ad9447ea19df245c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33869 } }
I20260927 12:40:53.411221 2712 consensus_queue.cc:260] T fe6108b3dfcd43d7b6063c49be1f9641 P af8e109d5d504a08aea237603b8b23a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8e109d5d504a08aea237603b8b23a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46773 } } peers { permanent_uuid: "6092fc28ff124c22aef46193200cd07c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40853 } } peers { permanent_uuid: "10b5013c5e0a4050ad9447ea19df245c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33869 } }
I20260927 12:40:53.411283 2710 raft_consensus.cc:374] T fe6108b3dfcd43d7b6063c49be1f9641 P 6092fc28ff124c22aef46193200cd07c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:40:53.411316 2710 raft_consensus.cc:729] T fe6108b3dfcd43d7b6063c49be1f9641 P 6092fc28ff124c22aef46193200cd07c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6092fc28ff124c22aef46193200cd07c, State: Initialized, Role: FOLLOWER
I20260927 12:40:53.411356 2711 tablet_bootstrap.cc:492] T fe6108b3dfcd43d7b6063c49be1f9641 P 10b5013c5e0a4050ad9447ea19df245c: No bootstrap required, opened a new log
I20260927 12:40:53.411437 2711 ts_tablet_manager.cc:1397] T fe6108b3dfcd43d7b6063c49be1f9641 P 10b5013c5e0a4050ad9447ea19df245c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 12:40:53.411432 2710 consensus_queue.cc:260] T fe6108b3dfcd43d7b6063c49be1f9641 P 6092fc28ff124c22aef46193200cd07c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8e109d5d504a08aea237603b8b23a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46773 } } peers { permanent_uuid: "6092fc28ff124c22aef46193200cd07c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40853 } } peers { permanent_uuid: "10b5013c5e0a4050ad9447ea19df245c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33869 } }
I20260927 12:40:53.411495 2712 ts_tablet_manager.cc:1428] T fe6108b3dfcd43d7b6063c49be1f9641 P af8e109d5d504a08aea237603b8b23a4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 12:40:53.411680 2710 ts_tablet_manager.cc:1428] T fe6108b3dfcd43d7b6063c49be1f9641 P 6092fc28ff124c22aef46193200cd07c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260927 12:40:53.411849 2711 raft_consensus.cc:348] T fe6108b3dfcd43d7b6063c49be1f9641 P 10b5013c5e0a4050ad9447ea19df245c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8e109d5d504a08aea237603b8b23a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46773 } } peers { permanent_uuid: "6092fc28ff124c22aef46193200cd07c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40853 } } peers { permanent_uuid: "10b5013c5e0a4050ad9447ea19df245c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33869 } }
I20260927 12:40:53.411952 2711 raft_consensus.cc:374] T fe6108b3dfcd43d7b6063c49be1f9641 P 10b5013c5e0a4050ad9447ea19df245c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:40:53.411983 2711 raft_consensus.cc:729] T fe6108b3dfcd43d7b6063c49be1f9641 P 10b5013c5e0a4050ad9447ea19df245c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10b5013c5e0a4050ad9447ea19df245c, State: Initialized, Role: FOLLOWER
I20260927 12:40:53.412096 2711 consensus_queue.cc:260] T fe6108b3dfcd43d7b6063c49be1f9641 P 10b5013c5e0a4050ad9447ea19df245c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8e109d5d504a08aea237603b8b23a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46773 } } peers { permanent_uuid: "6092fc28ff124c22aef46193200cd07c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40853 } } peers { permanent_uuid: "10b5013c5e0a4050ad9447ea19df245c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33869 } }
I20260927 12:40:53.412371 2711 ts_tablet_manager.cc:1428] T fe6108b3dfcd43d7b6063c49be1f9641 P 10b5013c5e0a4050ad9447ea19df245c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 12:40:53.535897 2716 raft_consensus.cc:482] T fe6108b3dfcd43d7b6063c49be1f9641 P 10b5013c5e0a4050ad9447ea19df245c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:40:53.536092 2716 raft_consensus.cc:504] T fe6108b3dfcd43d7b6063c49be1f9641 P 10b5013c5e0a4050ad9447ea19df245c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8e109d5d504a08aea237603b8b23a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46773 } } peers { permanent_uuid: "6092fc28ff124c22aef46193200cd07c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40853 } } peers { permanent_uuid: "10b5013c5e0a4050ad9447ea19df245c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33869 } }
I20260927 12:40:53.537134 2716 leader_election.cc:288] T fe6108b3dfcd43d7b6063c49be1f9641 P 10b5013c5e0a4050ad9447ea19df245c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af8e109d5d504a08aea237603b8b23a4 (127.0.98.1:46773), 6092fc28ff124c22aef46193200cd07c (127.0.98.2:40853)
I20260927 12:40:53.541074 2729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:53.537529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37891 (local address 127.0.98.1:46773)
0927 12:40:53.537841 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:53.537857 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:53.537875 (+ 18us) server_negotiation.cc:407] Connection header received
0927 12:40:53.537958 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:53.537963 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:53.538026 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:53.538120 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:40:53.538430 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:53.538995 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:53.540068 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:53.540501 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:53.540627 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:53.540692 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:53.540862 (+ 170us) server_negotiation.cc:299] Negotiation successful
0927 12:40:53.540933 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":112,"threads_started":1}
I20260927 12:40:53.541074 2728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:53.537441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46773 (local address 127.0.98.3:37891)
0927 12:40:53.537714 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0927 12:40:53.537736 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 12:40:53.537854 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:40:53.538156 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:40:53.538159 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:40:53.538175 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:40:53.538308 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:53.538313 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:53.539113 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:53.539118 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:53.539940 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:53.539948 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:53.540624 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:53.540628 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:53.540728 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:53.540750 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 12:40:53.540846 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 12:40:53.540917 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":100,"threads_started":1}
I20260927 12:40:53.541206 2720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:53.538056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40733 (local address 127.0.98.2:40853)
0927 12:40:53.538175 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:53.538183 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:53.538226 (+ 43us) server_negotiation.cc:407] Connection header received
0927 12:40:53.538379 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:53.538382 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:53.538430 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:53.538504 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:40:53.538767 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:53.539273 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:53.540247 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:53.540632 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:53.540744 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:53.540802 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:53.541025 (+ 223us) server_negotiation.cc:299] Negotiation successful
0927 12:40:53.541098 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 12:40:53.541222 2730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:53.537895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40853 (local address 127.0.98.3:40733)
0927 12:40:53.538148 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0927 12:40:53.538161 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:40:53.538267 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:40:53.538525 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:40:53.538528 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:40:53.538543 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:40:53.538664 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:53.538669 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:53.539408 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:53.539411 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:53.540130 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:53.540138 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:53.540755 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:53.540758 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:53.540853 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:53.540875 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 12:40:53.541007 (+ 132us) client_negotiation.cc:240] Negotiation successful
0927 12:40:53.541076 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":98,"threads_started":1}
I20260927 12:40:53.541669 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe6108b3dfcd43d7b6063c49be1f9641" candidate_uuid: "10b5013c5e0a4050ad9447ea19df245c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af8e109d5d504a08aea237603b8b23a4" is_pre_election: true
I20260927 12:40:53.541750 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe6108b3dfcd43d7b6063c49be1f9641" candidate_uuid: "10b5013c5e0a4050ad9447ea19df245c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6092fc28ff124c22aef46193200cd07c" is_pre_election: true
I20260927 12:40:53.541841 2279 raft_consensus.cc:2454] T fe6108b3dfcd43d7b6063c49be1f9641 P af8e109d5d504a08aea237603b8b23a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10b5013c5e0a4050ad9447ea19df245c in term 0.
I20260927 12:40:53.542006 2419 raft_consensus.cc:2454] T fe6108b3dfcd43d7b6063c49be1f9641 P 6092fc28ff124c22aef46193200cd07c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10b5013c5e0a4050ad9447ea19df245c in term 0.
I20260927 12:40:53.542164 2493 leader_election.cc:302] T fe6108b3dfcd43d7b6063c49be1f9641 P 10b5013c5e0a4050ad9447ea19df245c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b5013c5e0a4050ad9447ea19df245c, af8e109d5d504a08aea237603b8b23a4; no voters:
I20260927 12:40:53.542408 2716 raft_consensus.cc:2781] T fe6108b3dfcd43d7b6063c49be1f9641 P 10b5013c5e0a4050ad9447ea19df245c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:40:53.542490 2716 raft_consensus.cc:482] T fe6108b3dfcd43d7b6063c49be1f9641 P 10b5013c5e0a4050ad9447ea19df245c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:40:53.542546 2716 raft_consensus.cc:3037] T fe6108b3dfcd43d7b6063c49be1f9641 P 10b5013c5e0a4050ad9447ea19df245c [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:40:53.543514 2716 raft_consensus.cc:504] T fe6108b3dfcd43d7b6063c49be1f9641 P 10b5013c5e0a4050ad9447ea19df245c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8e109d5d504a08aea237603b8b23a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46773 } } peers { permanent_uuid: "6092fc28ff124c22aef46193200cd07c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40853 } } peers { permanent_uuid: "10b5013c5e0a4050ad9447ea19df245c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33869 } }
I20260927 12:40:53.543839 2716 leader_election.cc:288] T fe6108b3dfcd43d7b6063c49be1f9641 P 10b5013c5e0a4050ad9447ea19df245c [CANDIDATE]: Term 1 election: Requested vote from peers af8e109d5d504a08aea237603b8b23a4 (127.0.98.1:46773), 6092fc28ff124c22aef46193200cd07c (127.0.98.2:40853)
I20260927 12:40:53.544234 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe6108b3dfcd43d7b6063c49be1f9641" candidate_uuid: "10b5013c5e0a4050ad9447ea19df245c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6092fc28ff124c22aef46193200cd07c"
I20260927 12:40:53.544234 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe6108b3dfcd43d7b6063c49be1f9641" candidate_uuid: "10b5013c5e0a4050ad9447ea19df245c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af8e109d5d504a08aea237603b8b23a4"
I20260927 12:40:53.544348 2419 raft_consensus.cc:3037] T fe6108b3dfcd43d7b6063c49be1f9641 P 6092fc28ff124c22aef46193200cd07c [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:40:53.544378 2279 raft_consensus.cc:3037] T fe6108b3dfcd43d7b6063c49be1f9641 P af8e109d5d504a08aea237603b8b23a4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:40:53.545456 2419 raft_consensus.cc:2454] T fe6108b3dfcd43d7b6063c49be1f9641 P 6092fc28ff124c22aef46193200cd07c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10b5013c5e0a4050ad9447ea19df245c in term 1.
I20260927 12:40:53.545456 2279 raft_consensus.cc:2454] T fe6108b3dfcd43d7b6063c49be1f9641 P af8e109d5d504a08aea237603b8b23a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10b5013c5e0a4050ad9447ea19df245c in term 1.
I20260927 12:40:53.545732 2493 leader_election.cc:302] T fe6108b3dfcd43d7b6063c49be1f9641 P 10b5013c5e0a4050ad9447ea19df245c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10b5013c5e0a4050ad9447ea19df245c, af8e109d5d504a08aea237603b8b23a4; no voters:
I20260927 12:40:53.545892 2716 raft_consensus.cc:2781] T fe6108b3dfcd43d7b6063c49be1f9641 P 10b5013c5e0a4050ad9447ea19df245c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:40:53.546090 2716 raft_consensus.cc:686] T fe6108b3dfcd43d7b6063c49be1f9641 P 10b5013c5e0a4050ad9447ea19df245c [term 1 LEADER]: Becoming Leader. State: Replica: 10b5013c5e0a4050ad9447ea19df245c, State: Running, Role: LEADER
I20260927 12:40:53.546361 2716 consensus_queue.cc:237] T fe6108b3dfcd43d7b6063c49be1f9641 P 10b5013c5e0a4050ad9447ea19df245c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8e109d5d504a08aea237603b8b23a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46773 } } peers { permanent_uuid: "6092fc28ff124c22aef46193200cd07c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40853 } } peers { permanent_uuid: "10b5013c5e0a4050ad9447ea19df245c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33869 } }
I20260927 12:40:53.548553 2120 catalog_manager.cc:5495] T fe6108b3dfcd43d7b6063c49be1f9641 P 10b5013c5e0a4050ad9447ea19df245c reported cstate change: term changed from 0 to 1, leader changed from <none> to 10b5013c5e0a4050ad9447ea19df245c (127.0.98.3). New cstate: current_term: 1 leader_uuid: "10b5013c5e0a4050ad9447ea19df245c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af8e109d5d504a08aea237603b8b23a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6092fc28ff124c22aef46193200cd07c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "10b5013c5e0a4050ad9447ea19df245c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33869 } health_report { overall_health: HEALTHY } } }
I20260927 12:40:53.571559 2733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:53.568184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46842 (local address 127.0.98.3:33869)
0927 12:40:53.568473 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:53.568479 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:53.568491 (+ 12us) server_negotiation.cc:407] Connection header received
0927 12:40:53.568558 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:53.568560 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:53.568610 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:53.568714 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:40:53.568907 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:53.569460 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:53.570231 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:53.570486 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:53.570566 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:53.570786 (+ 220us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:40:53.571233 (+ 447us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:40:53.571346 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:53.571370 (+ 24us) server_negotiation.cc:299] Negotiation successful
0927 12:40:53.571415 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":146,"threads_started":1}
I20260927 12:40:53.581313 2720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:53.578531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45211 (local address 127.0.98.2:40853)
0927 12:40:53.578648 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:53.578652 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:53.578751 (+ 99us) server_negotiation.cc:407] Connection header received
0927 12:40:53.578855 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:53.578857 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:53.578913 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:53.578991 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:40:53.579214 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:53.579690 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:53.580476 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:53.580670 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:53.580741 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:53.580973 (+ 232us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:40:53.581035 (+ 62us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:40:53.581087 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:53.581161 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 12:40:53.581205 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 12:40:53.581393 2736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:53.578406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40853 (local address 127.0.98.3:45211)
0927 12:40:53.578675 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0927 12:40:53.578688 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:40:53.578772 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:40:53.579015 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:40:53.579017 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:40:53.579024 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 12:40:53.579122 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:53.579127 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:53.579773 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:53.579775 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:53.580383 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:53.580390 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:53.580757 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:53.580759 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:53.580827 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:53.580866 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 12:40:53.581103 (+ 237us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 12:40:53.581107 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 12:40:53.581155 (+ 48us) client_negotiation.cc:240] Negotiation successful
0927 12:40:53.581216 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":100,"threads_started":1}
I20260927 12:40:53.587132 2279 raft_consensus.cc:1261] T fe6108b3dfcd43d7b6063c49be1f9641 P af8e109d5d504a08aea237603b8b23a4 [term 1 FOLLOWER]: Refusing update from remote peer 10b5013c5e0a4050ad9447ea19df245c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:40:53.587313 2419 raft_consensus.cc:1261] T fe6108b3dfcd43d7b6063c49be1f9641 P 6092fc28ff124c22aef46193200cd07c [term 1 FOLLOWER]: Refusing update from remote peer 10b5013c5e0a4050ad9447ea19df245c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:40:53.587635 2716 consensus_queue.cc:1035] T fe6108b3dfcd43d7b6063c49be1f9641 P 10b5013c5e0a4050ad9447ea19df245c [LEADER]: Connected to new peer: Peer: permanent_uuid: "af8e109d5d504a08aea237603b8b23a4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:40:53.587870 2731 consensus_queue.cc:1035] T fe6108b3dfcd43d7b6063c49be1f9641 P 10b5013c5e0a4050ad9447ea19df245c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6092fc28ff124c22aef46193200cd07c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:40:53.605873 2733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:53.602340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39219 (local address 127.0.98.3:33869)
0927 12:40:53.602472 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:53.602476 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:53.602764 (+ 288us) server_negotiation.cc:407] Connection header received
0927 12:40:53.602857 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:53.602863 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:53.602922 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:53.602998 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:40:53.603271 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:53.603836 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:53.604654 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:53.604991 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:53.605109 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:53.605325 (+ 216us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:40:53.605408 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:40:53.605468 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:53.605665 (+ 197us) server_negotiation.cc:299] Negotiation successful
0927 12:40:53.605733 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 12:40:53.605891 2741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:53.602260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33869 (local address 127.0.98.2:39219)
0927 12:40:53.602658 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0927 12:40:53.602672 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:40:53.602769 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:40:53.603023 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:40:53.603026 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:40:53.603034 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 12:40:53.603155 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:53.603161 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:53.603943 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:53.603945 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:53.604538 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:53.604545 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:53.605106 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:53.605110 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:53.605186 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:53.605212 (+ 26us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 12:40:53.605506 (+ 294us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 12:40:53.605512 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 12:40:53.605646 (+ 134us) client_negotiation.cc:240] Negotiation successful
0927 12:40:53.605733 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":114,"threads_started":1}
W20260927 12:40:53.606859 2682 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 12:40:55.162842 2120 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:40:55.163055 2120 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:40:55.165472 2120 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3e9ae20299c2487da31d96a749ba1a33: Sending DeleteTablet for 3 replicas of tablet fe6108b3dfcd43d7b6063c49be1f9641
I20260927 12:40:55.166029 2259 tablet_service.cc:1514] Processing DeleteTablet for tablet fe6108b3dfcd43d7b6063c49be1f9641 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:40:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42600
I20260927 12:40:55.166064 2539 tablet_service.cc:1514] Processing DeleteTablet for tablet fe6108b3dfcd43d7b6063c49be1f9641 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:40:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47786
I20260927 12:40:55.166137 2398 tablet_service.cc:1514] Processing DeleteTablet for tablet fe6108b3dfcd43d7b6063c49be1f9641 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:40:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54582
I20260927 12:40:55.166443 2766 tablet_replica.cc:330] T fe6108b3dfcd43d7b6063c49be1f9641 P 6092fc28ff124c22aef46193200cd07c: stopping tablet replica
I20260927 12:40:55.166476 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 2194
I20260927 12:40:55.167258 2765 tablet_replica.cc:330] T fe6108b3dfcd43d7b6063c49be1f9641 P 10b5013c5e0a4050ad9447ea19df245c: stopping tablet replica
I20260927 12:40:55.167443 2765 raft_consensus.cc:2229] T fe6108b3dfcd43d7b6063c49be1f9641 P 10b5013c5e0a4050ad9447ea19df245c [term 1 LEADER]: Raft consensus shutting down.
I20260927 12:40:55.167753 2765 raft_consensus.cc:2258] T fe6108b3dfcd43d7b6063c49be1f9641 P 10b5013c5e0a4050ad9447ea19df245c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:40:55.168439 2766 raft_consensus.cc:2229] T fe6108b3dfcd43d7b6063c49be1f9641 P 6092fc28ff124c22aef46193200cd07c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:40:55.168586 2766 raft_consensus.cc:2258] T fe6108b3dfcd43d7b6063c49be1f9641 P 6092fc28ff124c22aef46193200cd07c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:40:55.169070 2765 ts_tablet_manager.cc:1905] T fe6108b3dfcd43d7b6063c49be1f9641 P 10b5013c5e0a4050ad9447ea19df245c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:40:55.169979 2766 ts_tablet_manager.cc:1905] T fe6108b3dfcd43d7b6063c49be1f9641 P 6092fc28ff124c22aef46193200cd07c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:40:55.171710 2765 ts_tablet_manager.cc:1918] T fe6108b3dfcd43d7b6063c49be1f9641 P 10b5013c5e0a4050ad9447ea19df245c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 12:40:55.171788 2765 log.cc:1196] T fe6108b3dfcd43d7b6063c49be1f9641 P 10b5013c5e0a4050ad9447ea19df245c: Deleting WAL directory at /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/minicluster-data/ts-2/wal/wals/fe6108b3dfcd43d7b6063c49be1f9641
I20260927 12:40:55.172070 2765 ts_tablet_manager.cc:1939] T fe6108b3dfcd43d7b6063c49be1f9641 P 10b5013c5e0a4050ad9447ea19df245c: Deleting consensus metadata
W20260927 12:40:55.172469 2109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46773 (error 108)
I20260927 12:40:55.172646 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 2334
I20260927 12:40:55.172659 2766 ts_tablet_manager.cc:1918] T fe6108b3dfcd43d7b6063c49be1f9641 P 6092fc28ff124c22aef46193200cd07c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 12:40:55.172727 2766 log.cc:1196] T fe6108b3dfcd43d7b6063c49be1f9641 P 6092fc28ff124c22aef46193200cd07c: Deleting WAL directory at /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790512840836255-392-0/minicluster-data/ts-1/wal/wals/fe6108b3dfcd43d7b6063c49be1f9641
I20260927 12:40:55.172971 2766 ts_tablet_manager.cc:1939] T fe6108b3dfcd43d7b6063c49be1f9641 P 6092fc28ff124c22aef46193200cd07c: Deleting consensus metadata
I20260927 12:40:55.173913 2107 catalog_manager.cc:4841] TS 10b5013c5e0a4050ad9447ea19df245c (127.0.98.3:33869): tablet fe6108b3dfcd43d7b6063c49be1f9641 (table test-table [id=c24af1bdc2534bd9a43e81da7292804b]) successfully deleted
I20260927 12:40:55.178565 2768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:55.178187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46773 (local address 127.0.0.1:56726)
0927 12:40:55.178422 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0927 12:40:55.178486 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":85,"threads_started":1}
W20260927 12:40:55.178803 2109 catalog_manager.cc:4584] TS af8e109d5d504a08aea237603b8b23a4 (127.0.98.1:46773): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fe6108b3dfcd43d7b6063c49be1f9641: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46773: connect: Connection refused (error 111)
I20260927 12:40:55.179916 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 2474
I20260927 12:40:55.180646 2768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:55.180042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40853 (local address 127.0.0.1:57294)
0927 12:40:55.180526 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0927 12:40:55.180575 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:40853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401}
I20260927 12:40:55.186884 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 2090
2026-09-27T12:40:55Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6392 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:40:55 dist-test-slave-bnc1 krb5kdc[2771](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:40:55 dist-test-slave-bnc1 krb5kdc[2771](info): set up 2 sockets
Sep 27 12:40:55 dist-test-slave-bnc1 krb5kdc[2771](info): commencing operation
krb5kdc: starting...
W20260927 12:40:56.225455 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/krb5kdc/test-user.keytab.
Sep 27 12:40:56 dist-test-slave-bnc1 krb5kdc[2771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:40:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:40:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:40:56.344169 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39773
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39773
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:40:56.453825 2784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:40:56.454101 2784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:40:56.454144 2784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:40:56.458170 2784 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:40:56.458240 2784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:40:56.458262 2784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:40:56.458282 2784 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:40:56.458300 2784 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:40:56.463727 2784 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39773
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:40:56.464811 2784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:40:56.465909 2784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 12:40:56.474370 2784 server_base.cc:1034] running on GCE node
W20260927 12:40:56.475391 2790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:40:56.475462 2789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:40:56.475391 2792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:40:56.476114 2784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:40:56.477229 2784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:40:56.478399 2784 hybrid_clock.cc:648] HybridClock initialized: now 1790512856478363 us; error 46 us; skew 500 ppm
Sep 27 12:40:56 dist-test-slave-bnc1 krb5kdc[2771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512856, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:40:56.488165 2784 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:40:56.489112 2784 webserver.cc:460] Webserver started at http://127.0.98.62:35599/ using document root <none> and password file <none>
I20260927 12:40:56.489773 2784 fs_manager.cc:359] Metadata directory not provided
I20260927 12:40:56.489835 2784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:40:56.490087 2784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:40:56.492130 2784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/minicluster-data/master-0/data/instance:
uuid: "5a832c90a2b14999b3ff5ce2f1423266"
format_stamp: "Formatted at 2026-09-27 12:40:56 on dist-test-slave-bnc1"
server_key: "b8a07e65247afd68b4e8dc9d97ca8ad6"
server_key_iv: "3c97c3a0cb7c4c753e8a5062824afe06"
server_key_version: "encryptionkey@0"
I20260927 12:40:56.492697 2784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/minicluster-data/master-0/wal/instance:
uuid: "5a832c90a2b14999b3ff5ce2f1423266"
format_stamp: "Formatted at 2026-09-27 12:40:56 on dist-test-slave-bnc1"
server_key: "b8a07e65247afd68b4e8dc9d97ca8ad6"
server_key_iv: "3c97c3a0cb7c4c753e8a5062824afe06"
server_key_version: "encryptionkey@0"
I20260927 12:40:56.496382 2784 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260927 12:40:56.498723 2799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:40:56.500059 2784 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 12:40:56.500190 2784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/minicluster-data/master-0/wal
uuid: "5a832c90a2b14999b3ff5ce2f1423266"
format_stamp: "Formatted at 2026-09-27 12:40:56 on dist-test-slave-bnc1"
server_key: "b8a07e65247afd68b4e8dc9d97ca8ad6"
server_key_iv: "3c97c3a0cb7c4c753e8a5062824afe06"
server_key_version: "encryptionkey@0"
I20260927 12:40:56.500298 2784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:40:56.517843 2784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:40:56.520604 2784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:40:56.520789 2784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:40:56.527606 2784 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39773
I20260927 12:40:56.527628 2851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39773 every 8 connection(s)
I20260927 12:40:56.528479 2784 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:40:56.530249 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 2784
I20260927 12:40:56.530333 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/minicluster-data/master-0/wal/instance
I20260927 12:40:56.530598 392 external_mini_cluster.cc:1428] Setting key 928a544f0e50d7429ec2f6b7bde0a0fc
I20260927 12:40:56.531178 2852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:40:56.536721 2852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a832c90a2b14999b3ff5ce2f1423266: Bootstrap starting.
Sep 27 12:40:56 dist-test-slave-bnc1 krb5kdc[2771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:40:56.539037 2852 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a832c90a2b14999b3ff5ce2f1423266: Neither blocks nor log segments found. Creating new log.
I20260927 12:40:56.539935 2852 log.cc:824] T 00000000000000000000000000000000 P 5a832c90a2b14999b3ff5ce2f1423266: Log is configured to *not* fsync() on all Append() calls
I20260927 12:40:56.542086 2852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a832c90a2b14999b3ff5ce2f1423266: No bootstrap required, opened a new log
I20260927 12:40:56.544029 2855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:56.531942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58964 (local address 127.0.98.62:39773)
0927 12:40:56.532354 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:56.532374 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:56.532407 (+ 33us) server_negotiation.cc:407] Connection header received
0927 12:40:56.533023 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:56.533040 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:56.533333 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:56.533675 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:40:56.534466 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:56.535420 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:56.536256 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:56.536725 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:56.536900 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:56.539270 (+ 2370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:40:56.539277 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:40:56.539295 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:40:56.539330 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:40:56.541157 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:56.541559 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:56.541581 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:56.541587 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:56.541716 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:56.541989 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:56.541992 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:56.541994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:56.542467 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:40:56.542647 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:56.542844 (+ 197us) server_negotiation.cc:299] Negotiation successful
0927 12:40:56.543119 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":147,"threads_started":1}
I20260927 12:40:56.546278 2852 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a832c90a2b14999b3ff5ce2f1423266 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a832c90a2b14999b3ff5ce2f1423266" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39773 } }
I20260927 12:40:56.546489 2852 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a832c90a2b14999b3ff5ce2f1423266 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:40:56.546530 2852 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a832c90a2b14999b3ff5ce2f1423266 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a832c90a2b14999b3ff5ce2f1423266, State: Initialized, Role: FOLLOWER
I20260927 12:40:56.546867 2852 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a832c90a2b14999b3ff5ce2f1423266 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a832c90a2b14999b3ff5ce2f1423266" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39773 } }
I20260927 12:40:56.546974 2852 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a832c90a2b14999b3ff5ce2f1423266 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:40:56.547016 2852 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a832c90a2b14999b3ff5ce2f1423266 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:40:56.547083 2852 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a832c90a2b14999b3ff5ce2f1423266 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:40:56.547844 2852 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a832c90a2b14999b3ff5ce2f1423266 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a832c90a2b14999b3ff5ce2f1423266" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39773 } }
I20260927 12:40:56.548116 2852 leader_election.cc:302] T 00000000000000000000000000000000 P 5a832c90a2b14999b3ff5ce2f1423266 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a832c90a2b14999b3ff5ce2f1423266; no voters:
I20260927 12:40:56.548347 2852 leader_election.cc:288] T 00000000000000000000000000000000 P 5a832c90a2b14999b3ff5ce2f1423266 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:40:56.548484 2857 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a832c90a2b14999b3ff5ce2f1423266 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:40:56.548714 2857 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a832c90a2b14999b3ff5ce2f1423266 [term 1 LEADER]: Becoming Leader. State: Replica: 5a832c90a2b14999b3ff5ce2f1423266, State: Running, Role: LEADER
I20260927 12:40:56.549031 2857 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a832c90a2b14999b3ff5ce2f1423266 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a832c90a2b14999b3ff5ce2f1423266" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39773 } }
I20260927 12:40:56.549502 2852 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a832c90a2b14999b3ff5ce2f1423266 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:40:56.550606 2859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a832c90a2b14999b3ff5ce2f1423266 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a832c90a2b14999b3ff5ce2f1423266. Latest consensus state: current_term: 1 leader_uuid: "5a832c90a2b14999b3ff5ce2f1423266" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a832c90a2b14999b3ff5ce2f1423266" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39773 } } }
I20260927 12:40:56.550745 2859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a832c90a2b14999b3ff5ce2f1423266 [sys.catalog]: This master's current role is: LEADER
I20260927 12:40:56.551146 2866 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:40:56.551116 2858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a832c90a2b14999b3ff5ce2f1423266 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a832c90a2b14999b3ff5ce2f1423266" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a832c90a2b14999b3ff5ce2f1423266" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39773 } } }
I20260927 12:40:56.551774 2858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a832c90a2b14999b3ff5ce2f1423266 [sys.catalog]: This master's current role is: LEADER
I20260927 12:40:56.554375 2866 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:40:56.559607 2866 catalog_manager.cc:1347] Generated new cluster ID: fe1491b1dd0c453c97beb2aaf4ff739d
I20260927 12:40:56.559672 2866 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:40:56.568933 2866 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:40:56.569934 2866 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:40:56.575306 2866 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a832c90a2b14999b3ff5ce2f1423266: Generated new TSK 0
I20260927 12:40:56.575901 2866 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 12:40:56.606345 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39773
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:40:56.721758 2878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:40:56.722044 2878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:40:56.722113 2878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:40:56.726424 2878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:40:56.726500 2878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:40:56.726584 2878 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:40:56.731877 2878 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39773
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2878
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:40:56.733000 2878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:40:56.734064 2878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:40:56.743432 2886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:40:56.743438 2883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:40:56.743431 2884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:40:56.743703 2878 server_base.cc:1034] running on GCE node
I20260927 12:40:56.744196 2878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:40:56.744817 2878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:40:56.745998 2878 hybrid_clock.cc:648] HybridClock initialized: now 1790512856745980 us; error 40 us; skew 500 ppm
Sep 27 12:40:56 dist-test-slave-bnc1 krb5kdc[2771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512856, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:40:56.748631 2878 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 12:40:56.749514 2878 webserver.cc:460] Webserver started at http://127.0.98.1:39483/ using document root <none> and password file <none>
I20260927 12:40:56.750168 2878 fs_manager.cc:359] Metadata directory not provided
I20260927 12:40:56.750255 2878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:40:56.750481 2878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:40:56.752516 2878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-0/data/instance:
uuid: "12487d17e48e438996fbb57c2a4fb420"
format_stamp: "Formatted at 2026-09-27 12:40:56 on dist-test-slave-bnc1"
server_key: "03010bbf7b425b36f306ee8ddf3206ba"
server_key_iv: "654e8188c575465b798c012b7e47c78e"
server_key_version: "encryptionkey@0"
I20260927 12:40:56.753116 2878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-0/wal/instance:
uuid: "12487d17e48e438996fbb57c2a4fb420"
format_stamp: "Formatted at 2026-09-27 12:40:56 on dist-test-slave-bnc1"
server_key: "03010bbf7b425b36f306ee8ddf3206ba"
server_key_iv: "654e8188c575465b798c012b7e47c78e"
server_key_version: "encryptionkey@0"
I20260927 12:40:56.756678 2878 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 12:40:56.759063 2893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:40:56.760288 2878 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 12:40:56.760434 2878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-0/wal
uuid: "12487d17e48e438996fbb57c2a4fb420"
format_stamp: "Formatted at 2026-09-27 12:40:56 on dist-test-slave-bnc1"
server_key: "03010bbf7b425b36f306ee8ddf3206ba"
server_key_iv: "654e8188c575465b798c012b7e47c78e"
server_key_version: "encryptionkey@0"
I20260927 12:40:56.760550 2878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:40:56.808987 2878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:40:56.812297 2878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:40:56.812505 2878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:40:56.813133 2878 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:40:56.814100 2878 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:40:56.814164 2878 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:40:56.814231 2878 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:40:56.814280 2878 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:40:56.823051 2878 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34383
I20260927 12:40:56.823089 3006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34383 every 8 connection(s)
I20260927 12:40:56.823994 2878 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:40:56.824139 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 2878
I20260927 12:40:56.824246 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-0/wal/instance
I20260927 12:40:56.824568 392 external_mini_cluster.cc:1428] Setting key 292b21955168711cd92cc4a7f5182c90
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Sep 27 12:40:56 dist-test-slave-bnc1 krb5kdc[2771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512856, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:40:56.841687 2855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:56.827150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41383 (local address 127.0.98.62:39773)
0927 12:40:56.827290 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:56.827296 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:56.828248 (+ 952us) server_negotiation.cc:407] Connection header received
0927 12:40:56.829636 (+ 1388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:56.829641 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:56.829700 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:56.829795 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:40:56.831011 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:56.832002 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:56.833111 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:56.833538 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:56.833712 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:56.837495 (+ 3783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:40:56.837503 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:40:56.837509 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:40:56.837545 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:40:56.839075 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:56.839695 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:56.839698 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:56.839699 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:56.839771 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:56.840184 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:56.840186 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:56.840187 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:56.840419 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:40:56.840496 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:56.841194 (+ 698us) server_negotiation.cc:299] Negotiation successful
0927 12:40:56.841385 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 12:40:56.842315 3010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:56.827634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39773 (local address 127.0.98.1:41383)
0927 12:40:56.828089 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0927 12:40:56.828141 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 12:40:56.829411 (+ 1270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:40:56.829954 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:40:56.829960 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:40:56.830368 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:40:56.830858 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:56.830871 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:56.832133 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:56.832135 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:56.832988 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:56.832995 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:56.833729 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:56.833732 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:56.833873 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:56.834535 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:40:56.834551 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:40:56.837330 (+ 2779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:40:56.839196 (+ 1866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:40:56.839206 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:40:56.839224 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:40:56.839600 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:40:56.839871 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:40:56.839877 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:40:56.839883 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:40:56.840089 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:40:56.840546 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:40:56.840553 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:40:56.840891 (+ 338us) client_negotiation.cc:769] Sending connection context
0927 12:40:56.841104 (+ 213us) client_negotiation.cc:240] Negotiation successful
0927 12:40:56.841393 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":133,"threads_started":1}
I20260927 12:40:56.843575 3007 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39773
I20260927 12:40:56.843801 3007 heartbeater.cc:464] Registering TS with master...
I20260927 12:40:56.844282 3007 heartbeater.cc:511] Master 127.0.98.62:39773 requested a full tablet report, sending...
I20260927 12:40:56.845889 2816 ts_manager.cc:194] Registered new tserver with Master: 12487d17e48e438996fbb57c2a4fb420 (127.0.98.1:34383)
I20260927 12:40:56.847097 2816 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41383
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 12:40:56.853417 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39773
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:40:56.965431 3012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:40:56.965756 3012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:40:56.965802 3012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:40:56.969967 3012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:40:56.970022 3012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:40:56.970081 3012 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:40:56.975437 3012 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39773
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3012
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:40:56.976502 3012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:40:56.977542 3012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:40:56.987295 3018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:40:56.987324 3017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:40:56.987596 3012 server_base.cc:1034] running on GCE node
W20260927 12:40:56.987320 3020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:40:56.988376 3012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:40:56.989169 3012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:40:56.990381 3012 hybrid_clock.cc:648] HybridClock initialized: now 1790512856990358 us; error 35 us; skew 500 ppm
Sep 27 12:40:56 dist-test-slave-bnc1 krb5kdc[2771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512856, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:40:56.993189 3012 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 12:40:56.994189 3012 webserver.cc:460] Webserver started at http://127.0.98.2:40239/ using document root <none> and password file <none>
I20260927 12:40:56.994752 3012 fs_manager.cc:359] Metadata directory not provided
I20260927 12:40:56.994841 3012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:40:56.995066 3012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:40:56.997097 3012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-1/data/instance:
uuid: "44143f68abb845d2b660115f2a0d4225"
format_stamp: "Formatted at 2026-09-27 12:40:56 on dist-test-slave-bnc1"
server_key: "b94250d570ee53ef1cfadd4ad8955afe"
server_key_iv: "f88e25d6dad1586e37514c268418dc74"
server_key_version: "encryptionkey@0"
I20260927 12:40:56.997730 3012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-1/wal/instance:
uuid: "44143f68abb845d2b660115f2a0d4225"
format_stamp: "Formatted at 2026-09-27 12:40:56 on dist-test-slave-bnc1"
server_key: "b94250d570ee53ef1cfadd4ad8955afe"
server_key_iv: "f88e25d6dad1586e37514c268418dc74"
server_key_version: "encryptionkey@0"
I20260927 12:40:57.001466 3012 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:40:57.004050 3027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:40:57.005247 3012 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:40:57.005386 3012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-1/wal
uuid: "44143f68abb845d2b660115f2a0d4225"
format_stamp: "Formatted at 2026-09-27 12:40:56 on dist-test-slave-bnc1"
server_key: "b94250d570ee53ef1cfadd4ad8955afe"
server_key_iv: "f88e25d6dad1586e37514c268418dc74"
server_key_version: "encryptionkey@0"
I20260927 12:40:57.005494 3012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:40:57.028877 3012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:40:57.031805 3012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:40:57.032015 3012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:40:57.032616 3012 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:40:57.033448 3012 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:40:57.033519 3012 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:40:57.033615 3012 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:40:57.033653 3012 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:40:57.042637 3012 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42599
I20260927 12:40:57.042663 3140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42599 every 8 connection(s)
I20260927 12:40:57.043589 3012 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:40:57.049953 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 3012
I20260927 12:40:57.050067 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-1/wal/instance
I20260927 12:40:57.050380 392 external_mini_cluster.cc:1428] Setting key 93687aff5ac479c536d0f760f2bf70d4
Sep 27 12:40:57 dist-test-slave-bnc1 krb5kdc[2771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512856, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:40:57.059190 2855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:57.045408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60819 (local address 127.0.98.62:39773)
0927 12:40:57.045586 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:57.045593 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:57.046496 (+ 903us) server_negotiation.cc:407] Connection header received
0927 12:40:57.047816 (+ 1320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:57.047818 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:57.047866 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:57.047955 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:40:57.049080 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:57.049781 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:57.051389 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:57.051736 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:57.051840 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:57.055508 (+ 3668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:40:57.055512 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:40:57.055514 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:40:57.055544 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:40:57.057063 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:57.057627 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:57.057634 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:57.057636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:57.057730 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:57.058087 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:57.058092 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:57.058094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:57.058280 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:40:57.058390 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:57.058901 (+ 511us) server_negotiation.cc:299] Negotiation successful
0927 12:40:57.059026 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 12:40:57.060071 3143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:57.045889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39773 (local address 127.0.98.2:60819)
0927 12:40:57.046337 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0927 12:40:57.046382 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 12:40:57.047609 (+ 1227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:40:57.048102 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:40:57.048108 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:40:57.048448 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:40:57.048923 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:57.048936 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:57.049913 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:57.049918 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:57.051265 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:57.051274 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:57.052072 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:57.052076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:57.052341 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:57.053061 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:40:57.053087 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:40:57.055317 (+ 2230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:40:57.057204 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:40:57.057210 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:40:57.057226 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:40:57.057496 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:40:57.057828 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:40:57.057832 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:40:57.057834 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:40:57.057982 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:40:57.058387 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:40:57.058394 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:40:57.058658 (+ 264us) client_negotiation.cc:769] Sending connection context
0927 12:40:57.058861 (+ 203us) client_negotiation.cc:240] Negotiation successful
0927 12:40:57.059217 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":127,"threads_started":1}
I20260927 12:40:57.061241 3141 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39773
I20260927 12:40:57.061493 3141 heartbeater.cc:464] Registering TS with master...
I20260927 12:40:57.062086 3141 heartbeater.cc:511] Master 127.0.98.62:39773 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 12:40:57.063243 2816 ts_manager.cc:194] Registered new tserver with Master: 44143f68abb845d2b660115f2a0d4225 (127.0.98.2:42599)
I20260927 12:40:57.063891 2816 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60819
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 12:40:57.078678 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39773
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:40:57.189137 3146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:40:57.189392 3146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:40:57.189438 3146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:40:57.193490 3146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:40:57.193554 3146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:40:57.193698 3146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:40:57.198963 3146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39773
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:40:57.199934 3146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:40:57.200913 3146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:40:57.210811 3152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:40:57.210800 3151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:40:57.210800 3154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:40:57.211004 3146 server_base.cc:1034] running on GCE node
I20260927 12:40:57.211408 3146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:40:57.212041 3146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:40:57.213200 3146 hybrid_clock.cc:648] HybridClock initialized: now 1790512857213183 us; error 29 us; skew 500 ppm
Sep 27 12:40:57 dist-test-slave-bnc1 krb5kdc[2771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512857, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:40:57.215983 3146 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 12:40:57.216938 3146 webserver.cc:460] Webserver started at http://127.0.98.3:44947/ using document root <none> and password file <none>
I20260927 12:40:57.217514 3146 fs_manager.cc:359] Metadata directory not provided
I20260927 12:40:57.217655 3146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:40:57.217903 3146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:40:57.219920 3146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-2/data/instance:
uuid: "77009446da5c498693432c0f32342b6e"
format_stamp: "Formatted at 2026-09-27 12:40:57 on dist-test-slave-bnc1"
server_key: "2c8343039d400f5f17766aad78517c69"
server_key_iv: "04f7f55607791e113885c991662306d7"
server_key_version: "encryptionkey@0"
I20260927 12:40:57.220501 3146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-2/wal/instance:
uuid: "77009446da5c498693432c0f32342b6e"
format_stamp: "Formatted at 2026-09-27 12:40:57 on dist-test-slave-bnc1"
server_key: "2c8343039d400f5f17766aad78517c69"
server_key_iv: "04f7f55607791e113885c991662306d7"
server_key_version: "encryptionkey@0"
I20260927 12:40:57.224282 3146 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 12:40:57.226805 3161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:40:57.227962 3146 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 12:40:57.228096 3146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-2/wal
uuid: "77009446da5c498693432c0f32342b6e"
format_stamp: "Formatted at 2026-09-27 12:40:57 on dist-test-slave-bnc1"
server_key: "2c8343039d400f5f17766aad78517c69"
server_key_iv: "04f7f55607791e113885c991662306d7"
server_key_version: "encryptionkey@0"
I20260927 12:40:57.228222 3146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:40:57.241680 3146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:40:57.244939 3146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:40:57.245134 3146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:40:57.245730 3146 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:40:57.246526 3146 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:40:57.246594 3146 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:40:57.246662 3146 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:40:57.246716 3146 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:40:57.256019 3146 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37213
I20260927 12:40:57.256038 3274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37213 every 8 connection(s)
I20260927 12:40:57.256891 3146 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:40:57.264930 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 3146
I20260927 12:40:57.265041 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-2/wal/instance
I20260927 12:40:57.265383 392 external_mini_cluster.cc:1428] Setting key 06a96929b76a25753d5c4087527b5643
Sep 27 12:40:57 dist-test-slave-bnc1 krb5kdc[2771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512857, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:40:57.270651 2855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:57.258816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49743 (local address 127.0.98.62:39773)
0927 12:40:57.258969 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:57.258973 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:57.259733 (+ 760us) server_negotiation.cc:407] Connection header received
0927 12:40:57.260987 (+ 1254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:57.260990 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:57.261036 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:57.261107 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:40:57.262231 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:57.262746 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:57.263952 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:57.264248 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:57.264368 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:57.267233 (+ 2865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:40:57.267236 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:40:57.267238 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:40:57.267260 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:40:57.268658 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:57.269183 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:57.269187 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:57.269188 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:57.269267 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:57.269625 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:57.269628 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:57.269630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:57.269804 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:40:57.269909 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:57.270373 (+ 464us) server_negotiation.cc:299] Negotiation successful
0927 12:40:57.270490 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 12:40:57.271390 3277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:57.259171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39773 (local address 127.0.98.3:49743)
0927 12:40:57.259586 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0927 12:40:57.259627 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 12:40:57.260802 (+ 1175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:40:57.261231 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:40:57.261237 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:40:57.261601 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:40:57.262095 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:57.262108 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:57.262875 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:57.262878 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:57.263843 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:40:57.263853 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:40:57.264427 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:40:57.264430 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:40:57.264595 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:57.265290 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:40:57.265314 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:40:57.267068 (+ 1754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:40:57.268789 (+ 1721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:40:57.268794 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:40:57.268809 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:40:57.269073 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:40:57.269365 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:40:57.269368 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:40:57.269371 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:40:57.269504 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:40:57.269906 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:40:57.269913 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:40:57.270149 (+ 236us) client_negotiation.cc:769] Sending connection context
0927 12:40:57.270322 (+ 173us) client_negotiation.cc:240] Negotiation successful
0927 12:40:57.270622 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":101,"threads_started":1}
I20260927 12:40:57.272591 3275 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39773
I20260927 12:40:57.272864 3275 heartbeater.cc:464] Registering TS with master...
I20260927 12:40:57.273422 3275 heartbeater.cc:511] Master 127.0.98.62:39773 requested a full tablet report, sending...
I20260927 12:40:57.274720 2816 ts_manager.cc:194] Registered new tserver with Master: 77009446da5c498693432c0f32342b6e (127.0.98.3:37213)
I20260927 12:40:57.275331 2816 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49743
I20260927 12:40:57.279047 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 12:40:57.293087 2855 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:40:57.291439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58976 (local address 127.0.98.62:39773)
0927 12:40:57.291637 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:57.291643 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:57.291797 (+ 154us) server_negotiation.cc:407] Connection header received
0927 12:40:57.291989 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:57.292001 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:57.292054 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:57.292161 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:40:57.292968 (+ 807us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58976: BlockingRecv error: recv got EOF from 127.0.0.1:58976 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 12:40:57.294190 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 2878
I20260927 12:40:57.299559 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 3012
I20260927 12:40:57.304939 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 3146
I20260927 12:40:57.310339 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 2784
2026-09-27T12:40:57Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2122 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790512840836255-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:40:57 dist-test-slave-bnc1 krb5kdc[3287](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:40:57 dist-test-slave-bnc1 krb5kdc[3287](info): set up 2 sockets
Sep 27 12:40:57 dist-test-slave-bnc1 krb5kdc[3287](info): commencing operation
krb5kdc: starting...
W20260927 12:40:58.349627 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790512840836255-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790512840836255-392-0/krb5kdc/test-user.keytab.
Sep 27 12:40:58 dist-test-slave-bnc1 krb5kdc[3287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:40:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:40:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:40:58.469030 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790512840836255-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790512840836255-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790512840836255-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41993
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41993
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:40:58.576999 3300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:40:58.577273 3300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:40:58.577313 3300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:40:58.581261 3300 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:40:58.581321 3300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:40:58.581339 3300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:40:58.581357 3300 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:40:58.581372 3300 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:40:58.586529 3300 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790512840836255-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790512840836255-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41993
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3300
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790512840836255-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:40:58.587541 3300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:40:58.588495 3300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:40:58.596879 3308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:40:58.596829 3306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:40:58.596949 3305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:40:58.597048 3300 server_base.cc:1034] running on GCE node
I20260927 12:40:58.597790 3300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:40:58.599011 3300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:40:58.600209 3300 hybrid_clock.cc:648] HybridClock initialized: now 1790512858600173 us; error 49 us; skew 500 ppm
Sep 27 12:40:58 dist-test-slave-bnc1 krb5kdc[3287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512858, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:40:58.603029 3300 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:40:58.603981 3300 webserver.cc:460] Webserver started at http://127.0.98.62:34309/ using document root <none> and password file <none>
I20260927 12:40:58.604549 3300 fs_manager.cc:359] Metadata directory not provided
I20260927 12:40:58.604640 3300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:40:58.604882 3300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:40:58.607884 3300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790512840836255-392-0/minicluster-data/master-0/data/instance:
uuid: "1b03480edf2b41f0a1ba65e9a66e9fd7"
format_stamp: "Formatted at 2026-09-27 12:40:58 on dist-test-slave-bnc1"
server_key: "1f11f8153dfdb426ca4e6f7dfd4e7132"
server_key_iv: "52baf06c29bbb38b3a1ec9b84a9a37f3"
server_key_version: "encryptionkey@0"
I20260927 12:40:58.608700 3300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790512840836255-392-0/minicluster-data/master-0/wal/instance:
uuid: "1b03480edf2b41f0a1ba65e9a66e9fd7"
format_stamp: "Formatted at 2026-09-27 12:40:58 on dist-test-slave-bnc1"
server_key: "1f11f8153dfdb426ca4e6f7dfd4e7132"
server_key_iv: "52baf06c29bbb38b3a1ec9b84a9a37f3"
server_key_version: "encryptionkey@0"
I20260927 12:40:58.613339 3300 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260927 12:40:58.615882 3315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:40:58.617096 3300 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:40:58.617270 3300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790512840836255-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790512840836255-392-0/minicluster-data/master-0/wal
uuid: "1b03480edf2b41f0a1ba65e9a66e9fd7"
format_stamp: "Formatted at 2026-09-27 12:40:58 on dist-test-slave-bnc1"
server_key: "1f11f8153dfdb426ca4e6f7dfd4e7132"
server_key_iv: "52baf06c29bbb38b3a1ec9b84a9a37f3"
server_key_version: "encryptionkey@0"
I20260927 12:40:58.617393 3300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790512840836255-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790512840836255-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790512840836255-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:40:58.629702 3300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:40:58.632614 3300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:40:58.632853 3300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:40:58.639703 3300 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41993
I20260927 12:40:58.639967 3367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41993 every 8 connection(s)
I20260927 12:40:58.640671 3300 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:40:58.643803 3368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:40:58.644680 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 3300
I20260927 12:40:58.644790 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790512840836255-392-0/minicluster-data/master-0/wal/instance
I20260927 12:40:58.645100 392 external_mini_cluster.cc:1428] Setting key 353bd23f17d79e0ce0644557d7645b18
I20260927 12:40:58.650156 3368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b03480edf2b41f0a1ba65e9a66e9fd7: Bootstrap starting.
I20260927 12:40:58.652715 3368 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b03480edf2b41f0a1ba65e9a66e9fd7: Neither blocks nor log segments found. Creating new log.
I20260927 12:40:58.653645 3368 log.cc:824] T 00000000000000000000000000000000 P 1b03480edf2b41f0a1ba65e9a66e9fd7: Log is configured to *not* fsync() on all Append() calls
Sep 27 12:40:58 dist-test-slave-bnc1 krb5kdc[3287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:40:58.655697 3368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b03480edf2b41f0a1ba65e9a66e9fd7: No bootstrap required, opened a new log
I20260927 12:40:58.659219 3368 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b03480edf2b41f0a1ba65e9a66e9fd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b03480edf2b41f0a1ba65e9a66e9fd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41993 } }
I20260927 12:40:58.659369 3371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:58.646420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53596 (local address 127.0.98.62:41993)
0927 12:40:58.646799 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:58.646820 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:58.646853 (+ 33us) server_negotiation.cc:407] Connection header received
0927 12:40:58.647471 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:58.647489 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:58.647821 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:58.648096 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:40:58.648948 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:58.649998 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:58.651113 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:58.651604 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:58.651721 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:58.654729 (+ 3008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:40:58.654733 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:40:58.654746 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:40:58.654774 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:40:58.656520 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:58.657015 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:58.657025 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:58.657030 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:58.657137 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:40:58.657427 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:40:58.657430 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:40:58.657432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:40:58.657864 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:40:58.658067 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:58.658363 (+ 296us) server_negotiation.cc:299] Negotiation successful
0927 12:40:58.658652 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":123,"threads_started":1}
I20260927 12:40:58.659451 3368 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b03480edf2b41f0a1ba65e9a66e9fd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:40:58.659520 3368 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b03480edf2b41f0a1ba65e9a66e9fd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b03480edf2b41f0a1ba65e9a66e9fd7, State: Initialized, Role: FOLLOWER
I20260927 12:40:58.659915 3368 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b03480edf2b41f0a1ba65e9a66e9fd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b03480edf2b41f0a1ba65e9a66e9fd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41993 } }
I20260927 12:40:58.660020 3368 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b03480edf2b41f0a1ba65e9a66e9fd7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:40:58.660058 3368 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b03480edf2b41f0a1ba65e9a66e9fd7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:40:58.660120 3368 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b03480edf2b41f0a1ba65e9a66e9fd7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:40:58.661178 3368 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b03480edf2b41f0a1ba65e9a66e9fd7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b03480edf2b41f0a1ba65e9a66e9fd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41993 } }
I20260927 12:40:58.661532 3368 leader_election.cc:302] T 00000000000000000000000000000000 P 1b03480edf2b41f0a1ba65e9a66e9fd7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b03480edf2b41f0a1ba65e9a66e9fd7; no voters:
I20260927 12:40:58.661926 3368 leader_election.cc:288] T 00000000000000000000000000000000 P 1b03480edf2b41f0a1ba65e9a66e9fd7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:40:58.662070 3373 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b03480edf2b41f0a1ba65e9a66e9fd7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:40:58.662283 3373 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b03480edf2b41f0a1ba65e9a66e9fd7 [term 1 LEADER]: Becoming Leader. State: Replica: 1b03480edf2b41f0a1ba65e9a66e9fd7, State: Running, Role: LEADER
I20260927 12:40:58.662310 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 12:40:58.662578 3373 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b03480edf2b41f0a1ba65e9a66e9fd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b03480edf2b41f0a1ba65e9a66e9fd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41993 } }
I20260927 12:40:58.663197 3368 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b03480edf2b41f0a1ba65e9a66e9fd7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:40:58.663640 3375 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b03480edf2b41f0a1ba65e9a66e9fd7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b03480edf2b41f0a1ba65e9a66e9fd7. Latest consensus state: current_term: 1 leader_uuid: "1b03480edf2b41f0a1ba65e9a66e9fd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b03480edf2b41f0a1ba65e9a66e9fd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41993 } } }
I20260927 12:40:58.663760 3375 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b03480edf2b41f0a1ba65e9a66e9fd7 [sys.catalog]: This master's current role is: LEADER
I20260927 12:40:58.664086 3374 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b03480edf2b41f0a1ba65e9a66e9fd7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b03480edf2b41f0a1ba65e9a66e9fd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b03480edf2b41f0a1ba65e9a66e9fd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41993 } } }
I20260927 12:40:58.664183 3374 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b03480edf2b41f0a1ba65e9a66e9fd7 [sys.catalog]: This master's current role is: LEADER
I20260927 12:40:58.664613 3377 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:40:58.667693 3377 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:40:58.673422 3377 catalog_manager.cc:1347] Generated new cluster ID: b05ee211bccc4b96a5c1f50cd23d48e2
I20260927 12:40:58.673488 3377 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:40:58.684929 3371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:40:58.679190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53612 (local address 127.0.98.62:41993)
0927 12:40:58.679300 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 12:40:58.679305 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:40:58.679836 (+ 531us) server_negotiation.cc:407] Connection header received
0927 12:40:58.679954 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:40:58.679959 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:40:58.680010 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:40:58.680096 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:40:58.680648 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:58.681428 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:58.682539 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:40:58.682935 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:40:58.683042 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:40:58.683911 (+ 869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:40:58.683915 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:40:58.683918 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:40:58.684497 (+ 579us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:40:58.684531 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:40:58.684551 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:40:58.684644 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 12:40:58.684722 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 12:40:58.684778 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 12:40:58.704548 3377 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:40:58.705519 3377 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:40:58.714067 3377 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1b03480edf2b41f0a1ba65e9a66e9fd7: Generated new TSK 0
I20260927 12:40:58.714980 3377 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:40:58.720641 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 3300
2026-09-27T12:40:58Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1408 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:40:58 dist-test-slave-bnc1 krb5kdc[3401](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:40:58 dist-test-slave-bnc1 krb5kdc[3401](info): set up 2 sockets
Sep 27 12:40:58 dist-test-slave-bnc1 krb5kdc[3401](info): commencing operation
krb5kdc: starting...
W20260927 12:40:59.777474 392 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/krb5kdc/test-user.keytab.
Sep 27 12:40:59 dist-test-slave-bnc1 krb5kdc[3401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512859, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:40:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:40:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:40:59.900561 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45367
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45367
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:41:00.010510 3414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:41:00.010777 3414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:41:00.010818 3414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:41:00.014806 3414 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:41:00.014861 3414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:41:00.014881 3414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:41:00.014897 3414 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:41:00.014912 3414 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:41:00.020016 3414 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45367
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3414
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:41:00.020992 3414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:41:00.021970 3414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:41:00.030448 3419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:00.030488 3414 server_base.cc:1034] running on GCE node
W20260927 12:41:00.030503 3422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:00.030779 3420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:00.031378 3414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:41:00.032564 3414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:41:00.033752 3414 hybrid_clock.cc:648] HybridClock initialized: now 1790512860033730 us; error 31 us; skew 500 ppm
Sep 27 12:41:00 dist-test-slave-bnc1 krb5kdc[3401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512860, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:41:00.036689 3414 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:41:00.037549 3414 webserver.cc:460] Webserver started at http://127.0.98.62:38441/ using document root <none> and password file <none>
I20260927 12:41:00.038151 3414 fs_manager.cc:359] Metadata directory not provided
I20260927 12:41:00.038211 3414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:41:00.038373 3414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:41:00.040478 3414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/minicluster-data/master-0/data/instance:
uuid: "f0b42adffad3410fb21a98e9b04a3465"
format_stamp: "Formatted at 2026-09-27 12:41:00 on dist-test-slave-bnc1"
server_key: "bcc338657ba0e790c94002e0622f4406"
server_key_iv: "1b20634cb4ae6efb20f6ff2e33c93ad3"
server_key_version: "encryptionkey@0"
I20260927 12:41:00.040988 3414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/minicluster-data/master-0/wal/instance:
uuid: "f0b42adffad3410fb21a98e9b04a3465"
format_stamp: "Formatted at 2026-09-27 12:41:00 on dist-test-slave-bnc1"
server_key: "bcc338657ba0e790c94002e0622f4406"
server_key_iv: "1b20634cb4ae6efb20f6ff2e33c93ad3"
server_key_version: "encryptionkey@0"
I20260927 12:41:00.044626 3414 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260927 12:41:00.047039 3429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:00.048312 3414 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 12:41:00.048451 3414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/minicluster-data/master-0/wal
uuid: "f0b42adffad3410fb21a98e9b04a3465"
format_stamp: "Formatted at 2026-09-27 12:41:00 on dist-test-slave-bnc1"
server_key: "bcc338657ba0e790c94002e0622f4406"
server_key_iv: "1b20634cb4ae6efb20f6ff2e33c93ad3"
server_key_version: "encryptionkey@0"
I20260927 12:41:00.048560 3414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:41:00.066329 3414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:41:00.069594 3414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:41:00.069901 3414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:41:00.077037 3414 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45367
I20260927 12:41:00.077399 3481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45367 every 8 connection(s)
I20260927 12:41:00.078089 3414 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:41:00.080922 3482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:41:00.086614 3482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0b42adffad3410fb21a98e9b04a3465: Bootstrap starting.
I20260927 12:41:00.086833 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 3414
I20260927 12:41:00.086987 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/minicluster-data/master-0/wal/instance
I20260927 12:41:00.087260 392 external_mini_cluster.cc:1428] Setting key 96e9124f518acdbae36a28ca48056e2c
I20260927 12:41:00.089270 3482 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0b42adffad3410fb21a98e9b04a3465: Neither blocks nor log segments found. Creating new log.
I20260927 12:41:00.090058 3482 log.cc:824] T 00000000000000000000000000000000 P f0b42adffad3410fb21a98e9b04a3465: Log is configured to *not* fsync() on all Append() calls
I20260927 12:41:00.091892 3482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0b42adffad3410fb21a98e9b04a3465: No bootstrap required, opened a new log
I20260927 12:41:00.095466 3482 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0b42adffad3410fb21a98e9b04a3465 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0b42adffad3410fb21a98e9b04a3465" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45367 } }
I20260927 12:41:00.095716 3482 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0b42adffad3410fb21a98e9b04a3465 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:41:00.095799 3482 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0b42adffad3410fb21a98e9b04a3465 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0b42adffad3410fb21a98e9b04a3465, State: Initialized, Role: FOLLOWER
Sep 27 12:41:00 dist-test-slave-bnc1 krb5kdc[3401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512859, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:41:00.096334 3482 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0b42adffad3410fb21a98e9b04a3465 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0b42adffad3410fb21a98e9b04a3465" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45367 } }
I20260927 12:41:00.096522 3482 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0b42adffad3410fb21a98e9b04a3465 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:41:00.096601 3482 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0b42adffad3410fb21a98e9b04a3465 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:41:00.096695 3482 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0b42adffad3410fb21a98e9b04a3465 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:41:00.097926 3482 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0b42adffad3410fb21a98e9b04a3465 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0b42adffad3410fb21a98e9b04a3465" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45367 } }
I20260927 12:41:00.098300 3482 leader_election.cc:302] T 00000000000000000000000000000000 P f0b42adffad3410fb21a98e9b04a3465 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0b42adffad3410fb21a98e9b04a3465; no voters:
I20260927 12:41:00.098630 3482 leader_election.cc:288] T 00000000000000000000000000000000 P f0b42adffad3410fb21a98e9b04a3465 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:41:00.098769 3487 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0b42adffad3410fb21a98e9b04a3465 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:41:00.098997 3487 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0b42adffad3410fb21a98e9b04a3465 [term 1 LEADER]: Becoming Leader. State: Replica: f0b42adffad3410fb21a98e9b04a3465, State: Running, Role: LEADER
I20260927 12:41:00.099359 3487 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0b42adffad3410fb21a98e9b04a3465 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0b42adffad3410fb21a98e9b04a3465" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45367 } }
I20260927 12:41:00.100106 3482 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0b42adffad3410fb21a98e9b04a3465 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:41:00.100625 3487 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0b42adffad3410fb21a98e9b04a3465 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0b42adffad3410fb21a98e9b04a3465. Latest consensus state: current_term: 1 leader_uuid: "f0b42adffad3410fb21a98e9b04a3465" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0b42adffad3410fb21a98e9b04a3465" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45367 } } }
I20260927 12:41:00.100744 3487 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0b42adffad3410fb21a98e9b04a3465 [sys.catalog]: This master's current role is: LEADER
I20260927 12:41:00.101080 3489 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0b42adffad3410fb21a98e9b04a3465 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f0b42adffad3410fb21a98e9b04a3465" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0b42adffad3410fb21a98e9b04a3465" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45367 } } }
I20260927 12:41:00.101171 3489 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0b42adffad3410fb21a98e9b04a3465 [sys.catalog]: This master's current role is: LEADER
I20260927 12:41:00.103255 3485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:00.088562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40396 (local address 127.0.98.62:45367)
0927 12:41:00.088985 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:00.089000 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:00.089033 (+ 33us) server_negotiation.cc:407] Connection header received
0927 12:41:00.089756 (+ 723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:00.089775 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:00.090111 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:00.090481 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:00.091294 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:00.092328 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:00.093179 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:00.093729 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:00.093905 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:00.096860 (+ 2955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:00.096865 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:00.096880 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:41:00.096909 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:00.099131 (+ 2222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:00.099723 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:00.099729 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:00.099736 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:00.099883 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:00.100247 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:00.100250 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:00.100252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:00.100783 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:00.101670 (+ 887us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:00.101855 (+ 185us) server_negotiation.cc:299] Negotiation successful
0927 12:41:00.102187 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":117,"threads_started":1}
W20260927 12:41:00.105243 3502 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f0b42adffad3410fb21a98e9b04a3465: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 12:41:00.105314 3502 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 12:41:00.105384 3492 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:41:00.106241 3492 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:41:00.111574 3492 catalog_manager.cc:1347] Generated new cluster ID: 9202f2f7a48845eb83a23a71665a3676
I20260927 12:41:00.111639 3492 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:41:00.128858 3492 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:41:00.129899 3492 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:41:00.139763 3492 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f0b42adffad3410fb21a98e9b04a3465: Generated new TSK 0
I20260927 12:41:00.140599 3492 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 12:41:00.177115 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45367
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:41:00.288193 3508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:41:00.288506 3508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:41:00.288584 3508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:41:00.292651 3508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:41:00.292729 3508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:41:00.292824 3508 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:41:00.298054 3508 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3508
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:41:00.299113 3508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:41:00.300220 3508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:41:00.310675 3513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:00.310690 3516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:00.310891 3508 server_base.cc:1034] running on GCE node
W20260927 12:41:00.310920 3514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:00.311362 3508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:41:00.311983 3508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:41:00.313180 3508 hybrid_clock.cc:648] HybridClock initialized: now 1790512860313145 us; error 47 us; skew 500 ppm
Sep 27 12:41:00 dist-test-slave-bnc1 krb5kdc[3401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512860, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:41:00.315903 3508 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 12:41:00.316779 3508 webserver.cc:460] Webserver started at http://127.0.98.1:40511/ using document root <none> and password file <none>
I20260927 12:41:00.317329 3508 fs_manager.cc:359] Metadata directory not provided
I20260927 12:41:00.317420 3508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:41:00.317696 3508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:41:00.319712 3508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-0/data/instance:
uuid: "ce3ce9f0a6464123b1e2597e00a9643c"
format_stamp: "Formatted at 2026-09-27 12:41:00 on dist-test-slave-bnc1"
server_key: "6affaac4bd66a3229adde1e181ebcdaa"
server_key_iv: "c0b3b4796943dd8288a150eb864bea16"
server_key_version: "encryptionkey@0"
I20260927 12:41:00.320291 3508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ce3ce9f0a6464123b1e2597e00a9643c"
format_stamp: "Formatted at 2026-09-27 12:41:00 on dist-test-slave-bnc1"
server_key: "6affaac4bd66a3229adde1e181ebcdaa"
server_key_iv: "c0b3b4796943dd8288a150eb864bea16"
server_key_version: "encryptionkey@0"
I20260927 12:41:00.324208 3508 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 12:41:00.326560 3523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:00.327687 3508 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 12:41:00.327812 3508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-0/wal
uuid: "ce3ce9f0a6464123b1e2597e00a9643c"
format_stamp: "Formatted at 2026-09-27 12:41:00 on dist-test-slave-bnc1"
server_key: "6affaac4bd66a3229adde1e181ebcdaa"
server_key_iv: "c0b3b4796943dd8288a150eb864bea16"
server_key_version: "encryptionkey@0"
I20260927 12:41:00.327916 3508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:41:00.348959 3508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:41:00.352433 3508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:41:00.352638 3508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:41:00.353287 3508 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:41:00.354207 3508 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:41:00.354278 3508 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:00.354343 3508 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:41:00.354393 3508 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:00.363379 3508 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42901
I20260927 12:41:00.363406 3636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42901 every 8 connection(s)
I20260927 12:41:00.364318 3508 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:41:00.373875 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 3508
I20260927 12:41:00.374006 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-0/wal/instance
Sep 27 12:41:00 dist-test-slave-bnc1 krb5kdc[3401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512860, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:41:00.374305 392 external_mini_cluster.cc:1428] Setting key 40d580ee974c8908b0f7cbcbabc1e780
I20260927 12:41:00.378780 3485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:00.366256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47991 (local address 127.0.98.62:45367)
0927 12:41:00.366407 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:00.366412 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:00.367344 (+ 932us) server_negotiation.cc:407] Connection header received
0927 12:41:00.368753 (+ 1409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:00.368756 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:00.368802 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:00.368876 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:00.370198 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:00.370731 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:00.371932 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:00.372161 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:00.372259 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:00.374913 (+ 2654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:00.374916 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:00.374919 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:41:00.374953 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:00.376660 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:00.377168 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:00.377175 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:00.377180 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:00.377276 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:00.377669 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:00.377673 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:00.377675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:00.377863 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:00.377973 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:00.378494 (+ 521us) server_negotiation.cc:299] Negotiation successful
0927 12:41:00.378617 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 12:41:00.379665 3639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:00.366725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45367 (local address 127.0.98.1:47991)
0927 12:41:00.367179 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:00.367222 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:00.368479 (+ 1257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:00.369077 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:00.369085 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:00.369458 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:00.370042 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:00.370056 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:00.370841 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:00.370844 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:00.371735 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:00.371743 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:00.372301 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:00.372303 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:00.372444 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:00.373012 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:00.373030 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:00.374738 (+ 1708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:41:00.376789 (+ 2051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:00.376794 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:00.376807 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:00.377069 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:00.377370 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:00.377373 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:00.377374 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:00.377547 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:00.377981 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:41:00.377990 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:41:00.378251 (+ 261us) client_negotiation.cc:769] Sending connection context
0927 12:41:00.378437 (+ 186us) client_negotiation.cc:240] Negotiation successful
0927 12:41:00.378823 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":114,"threads_started":1}
I20260927 12:41:00.381218 3637 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45367
I20260927 12:41:00.381511 3637 heartbeater.cc:464] Registering TS with master...
I20260927 12:41:00.382045 3637 heartbeater.cc:511] Master 127.0.98.62:45367 requested a full tablet report, sending...
I20260927 12:41:00.383776 3446 ts_manager.cc:194] Registered new tserver with Master: ce3ce9f0a6464123b1e2597e00a9643c (127.0.98.1:42901)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 12:41:00.385272 3446 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47991
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 12:41:00.402555 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45367
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:41:00.512684 3642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:41:00.512970 3642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:41:00.513053 3642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:41:00.517158 3642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:41:00.517215 3642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:41:00.517328 3642 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:41:00.522678 3642 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3642
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:41:00.523802 3642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:41:00.524863 3642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:41:00.534278 3647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:00.534354 3650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:00.534363 3642 server_base.cc:1034] running on GCE node
W20260927 12:41:00.534283 3648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:00.534953 3642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:41:00.535576 3642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:41:00.536760 3642 hybrid_clock.cc:648] HybridClock initialized: now 1790512860536743 us; error 25 us; skew 500 ppm
Sep 27 12:41:00 dist-test-slave-bnc1 krb5kdc[3401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512860, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:41:00.539441 3642 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 12:41:00.540323 3642 webserver.cc:460] Webserver started at http://127.0.98.2:40603/ using document root <none> and password file <none>
I20260927 12:41:00.540881 3642 fs_manager.cc:359] Metadata directory not provided
I20260927 12:41:00.540971 3642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:41:00.541189 3642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:41:00.543226 3642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-1/data/instance:
uuid: "46aa303fa424419cada22bc9275a4bae"
format_stamp: "Formatted at 2026-09-27 12:41:00 on dist-test-slave-bnc1"
server_key: "b5b1da500ddb9fd348a5db8da47136b2"
server_key_iv: "4589a00ffdf69ff14f392976b3989975"
server_key_version: "encryptionkey@0"
I20260927 12:41:00.543782 3642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-1/wal/instance:
uuid: "46aa303fa424419cada22bc9275a4bae"
format_stamp: "Formatted at 2026-09-27 12:41:00 on dist-test-slave-bnc1"
server_key: "b5b1da500ddb9fd348a5db8da47136b2"
server_key_iv: "4589a00ffdf69ff14f392976b3989975"
server_key_version: "encryptionkey@0"
I20260927 12:41:00.547487 3642 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260927 12:41:00.550654 3657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:00.552551 3642 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 12:41:00.552677 3642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-1/wal
uuid: "46aa303fa424419cada22bc9275a4bae"
format_stamp: "Formatted at 2026-09-27 12:41:00 on dist-test-slave-bnc1"
server_key: "b5b1da500ddb9fd348a5db8da47136b2"
server_key_iv: "4589a00ffdf69ff14f392976b3989975"
server_key_version: "encryptionkey@0"
I20260927 12:41:00.552822 3642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:41:00.575948 3642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:41:00.580360 3642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:41:00.580590 3642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:41:00.581408 3642 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:41:00.582659 3642 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:41:00.582731 3642 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:00.582794 3642 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:41:00.582845 3642 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:00.596452 3770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41111 every 8 connection(s)
I20260927 12:41:00.596477 3642 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41111
I20260927 12:41:00.597926 3642 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:41:00.598835 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 3642
I20260927 12:41:00.598989 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-1/wal/instance
I20260927 12:41:00.599265 392 external_mini_cluster.cc:1428] Setting key 9f9bf07a27f1b5f9628ff1a78e5b1c98
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Sep 27 12:41:00 dist-test-slave-bnc1 krb5kdc[3401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512860, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:41:00.621836 3485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:00.601521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49943 (local address 127.0.98.62:45367)
0927 12:41:00.601683 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:00.601690 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:00.603734 (+ 2044us) server_negotiation.cc:407] Connection header received
0927 12:41:00.605185 (+ 1451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:00.605188 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:00.605249 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:00.605335 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:00.607140 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:00.608010 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:00.609926 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:00.610235 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:00.610361 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:00.617724 (+ 7363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:00.617728 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:00.617731 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:41:00.617762 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:00.619473 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:00.620103 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:00.620106 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:00.620108 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:00.620188 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:00.620553 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:00.620556 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:00.620558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:00.620763 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:00.620837 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:00.621489 (+ 652us) server_negotiation.cc:299] Negotiation successful
0927 12:41:00.621667 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 12:41:00.623126 3773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:00.601918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45367 (local address 127.0.98.2:49943)
0927 12:41:00.603576 (+ 1658us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:00.603621 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:00.604971 (+ 1350us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:00.605558 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:00.605571 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:00.606172 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:00.606947 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:00.606973 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:00.608166 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:00.608171 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:00.609757 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:00.609772 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:00.610424 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:00.610427 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:00.610657 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:00.611438 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:00.611464 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:00.614115 (+ 2651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:41:00.619614 (+ 5499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:00.619620 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:00.619638 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:00.619988 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:00.620289 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:00.620291 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:00.620293 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:00.620455 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:00.620911 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:41:00.620917 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:41:00.621219 (+ 302us) client_negotiation.cc:769] Sending connection context
0927 12:41:00.622075 (+ 856us) client_negotiation.cc:240] Negotiation successful
0927 12:41:00.622358 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1469,"thread_start_us":134,"threads_started":1}
I20260927 12:41:00.624321 3771 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45367
I20260927 12:41:00.624588 3771 heartbeater.cc:464] Registering TS with master...
I20260927 12:41:00.625167 3771 heartbeater.cc:511] Master 127.0.98.62:45367 requested a full tablet report, sending...
I20260927 12:41:00.626385 3446 ts_manager.cc:194] Registered new tserver with Master: 46aa303fa424419cada22bc9275a4bae (127.0.98.2:41111)
I20260927 12:41:00.626984 3446 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49943
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 12:41:00.634688 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45367
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:41:00.746232 3776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:41:00.746488 3776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:41:00.746529 3776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:41:00.750523 3776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:41:00.750581 3776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:41:00.750660 3776 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:41:00.755995 3776 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:41:00.757062 3776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:41:00.758116 3776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:41:00.767490 3782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:00.767555 3776 server_base.cc:1034] running on GCE node
W20260927 12:41:00.768635 3784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:00.769009 3781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:00.769436 3776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:41:00.770141 3776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:41:00.771351 3776 hybrid_clock.cc:648] HybridClock initialized: now 1790512860771308 us; error 52 us; skew 500 ppm
Sep 27 12:41:00 dist-test-slave-bnc1 krb5kdc[3401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512860, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:41:00.774595 3776 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 12:41:00.775417 3776 webserver.cc:460] Webserver started at http://127.0.98.3:34787/ using document root <none> and password file <none>
I20260927 12:41:00.775934 3776 fs_manager.cc:359] Metadata directory not provided
I20260927 12:41:00.775991 3776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:41:00.776147 3776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:41:00.778467 3776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-2/data/instance:
uuid: "06a99d8aa029422cb11c1da1b3506d97"
format_stamp: "Formatted at 2026-09-27 12:41:00 on dist-test-slave-bnc1"
server_key: "c572b47ffe18886206439a6c0079d5a4"
server_key_iv: "f25c65b6f6ffd0d2e6d1ec8d903f9fc2"
server_key_version: "encryptionkey@0"
I20260927 12:41:00.778990 3776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-2/wal/instance:
uuid: "06a99d8aa029422cb11c1da1b3506d97"
format_stamp: "Formatted at 2026-09-27 12:41:00 on dist-test-slave-bnc1"
server_key: "c572b47ffe18886206439a6c0079d5a4"
server_key_iv: "f25c65b6f6ffd0d2e6d1ec8d903f9fc2"
server_key_version: "encryptionkey@0"
I20260927 12:41:00.782691 3776 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 12:41:00.785034 3791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:00.786286 3776 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260927 12:41:00.786386 3776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-2/wal
uuid: "06a99d8aa029422cb11c1da1b3506d97"
format_stamp: "Formatted at 2026-09-27 12:41:00 on dist-test-slave-bnc1"
server_key: "c572b47ffe18886206439a6c0079d5a4"
server_key_iv: "f25c65b6f6ffd0d2e6d1ec8d903f9fc2"
server_key_version: "encryptionkey@0"
I20260927 12:41:00.786473 3776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:41:00.807875 3776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:41:00.810724 3776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:41:00.810875 3776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:41:00.811491 3776 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:41:00.812302 3776 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:41:00.812347 3776 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:00.812415 3776 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:41:00.812456 3776 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:00.820977 3776 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33557
I20260927 12:41:00.820996 3904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33557 every 8 connection(s)
I20260927 12:41:00.821928 3776 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:41:00.831427 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 3776
Sep 27 12:41:00 dist-test-slave-bnc1 krb5kdc[3401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512860, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:41:00.831565 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790512840836255-392-0/minicluster-data/ts-2/wal/instance
I20260927 12:41:00.831851 392 external_mini_cluster.cc:1428] Setting key ef589e55d432a2482c69b0462a53ff8e
I20260927 12:41:00.835940 3485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:00.823723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41501 (local address 127.0.98.62:45367)
0927 12:41:00.823863 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:00.823867 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:00.824693 (+ 826us) server_negotiation.cc:407] Connection header received
0927 12:41:00.826106 (+ 1413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:00.826109 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:00.826163 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:00.826264 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:00.827410 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:00.828040 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:00.829185 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:00.829407 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:00.829527 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:00.832293 (+ 2766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:00.832297 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:00.832300 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:41:00.832331 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:00.833997 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:00.834609 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:00.834612 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:00.834613 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:00.834690 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:00.835003 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:00.835006 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:00.835007 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:00.835177 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:00.835254 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:00.835724 (+ 470us) server_negotiation.cc:299] Negotiation successful
0927 12:41:00.835824 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 12:41:00.836808 3907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:00.824117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45367 (local address 127.0.98.3:41501)
0927 12:41:00.824545 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:00.824585 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:00.825880 (+ 1295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:00.826434 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:00.826441 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:00.826792 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:00.827261 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:00.827274 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:00.828161 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:00.828165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:00.829062 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:00.829070 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:00.829557 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:00.829562 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:00.829764 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:00.830363 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:00.830380 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:00.832097 (+ 1717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:41:00.834117 (+ 2020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:00.834123 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:00.834139 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:00.834521 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:00.834771 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:00.834773 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:00.834774 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:00.834908 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:00.835264 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:41:00.835269 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:41:00.835521 (+ 252us) client_negotiation.cc:769] Sending connection context
0927 12:41:00.835682 (+ 161us) client_negotiation.cc:240] Negotiation successful
0927 12:41:00.835939 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":107,"threads_started":1}
I20260927 12:41:00.837996 3905 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45367
I20260927 12:41:00.838253 3905 heartbeater.cc:464] Registering TS with master...
I20260927 12:41:00.839169 3905 heartbeater.cc:511] Master 127.0.98.62:45367 requested a full tablet report, sending...
I20260927 12:41:00.840334 3446 ts_manager.cc:194] Registered new tserver with Master: 06a99d8aa029422cb11c1da1b3506d97 (127.0.98.3:33557)
I20260927 12:41:00.840971 3446 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41501
I20260927 12:41:00.845688 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 12:41:00 dist-test-slave-bnc1 krb5kdc[3401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512860, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 27 12:41:00 dist-test-slave-bnc1 krb5kdc[3401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512860, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:41:00.870395 3485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:00.862586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40400 (local address 127.0.98.62:45367)
0927 12:41:00.862721 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:00.862725 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:00.862867 (+ 142us) server_negotiation.cc:407] Connection header received
0927 12:41:00.863021 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:00.863025 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:00.863071 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:00.863169 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:00.863815 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:00.864334 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:00.865129 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:00.865383 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:00.865475 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:00.867782 (+ 2307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:00.867785 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:00.867787 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:41:00.867810 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:00.868990 (+ 1180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:00.869348 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:00.869350 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:00.869351 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:00.869418 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:00.869679 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:00.869681 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:00.869682 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:00.869877 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:00.869979 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:00.870186 (+ 207us) server_negotiation.cc:299] Negotiation successful
0927 12:41:00.870276 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260927 12:41:00.870795 3446 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:40400
I20260927 12:41:00.871758 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 3508
I20260927 12:41:00.877295 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 3642
I20260927 12:41:00.882364 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 3776
I20260927 12:41:00.887442 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 3414
2026-09-27T12:41:00Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2168 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:41:00 dist-test-slave-bnc1 krb5kdc[3917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:41:00 dist-test-slave-bnc1 krb5kdc[3917](info): set up 2 sockets
Sep 27 12:41:00 dist-test-slave-bnc1 krb5kdc[3917](info): commencing operation
krb5kdc: starting...
W20260927 12:41:01.925733 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/krb5kdc/test-user.keytab.
Sep 27 12:41:02 dist-test-slave-bnc1 krb5kdc[3917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:41:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:41:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:41:02.044520 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39743
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39743
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:41:02.152585 3930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:41:02.152824 3930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:41:02.152863 3930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:41:02.156893 3930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:41:02.156949 3930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:41:02.156968 3930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:41:02.156985 3930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:41:02.157045 3930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:41:02.162160 3930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39743
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:41:02.163168 3930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:41:02.164157 3930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:41:02.172741 3938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:02.172801 3936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:02.172755 3935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:02.172945 3930 server_base.cc:1034] running on GCE node
I20260927 12:41:02.173906 3930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:41:02.174986 3930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:41:02.176167 3930 hybrid_clock.cc:648] HybridClock initialized: now 1790512862176154 us; error 27 us; skew 500 ppm
Sep 27 12:41:02 dist-test-slave-bnc1 krb5kdc[3917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512862, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:41:02.183225 3930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:41:02.184101 3930 webserver.cc:460] Webserver started at http://127.0.98.62:39935/ using document root <none> and password file <none>
I20260927 12:41:02.184648 3930 fs_manager.cc:359] Metadata directory not provided
I20260927 12:41:02.184707 3930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:41:02.184937 3930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:41:02.186996 3930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/minicluster-data/master-0/data/instance:
uuid: "665a7da23a874dc68f68eea1fe3d44be"
format_stamp: "Formatted at 2026-09-27 12:41:02 on dist-test-slave-bnc1"
server_key: "2c5d8d03fac2e5d61e2da4c41ba8a5f5"
server_key_iv: "d2541055ead98b619e5bc46fce37ef3c"
server_key_version: "encryptionkey@0"
I20260927 12:41:02.187559 3930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/minicluster-data/master-0/wal/instance:
uuid: "665a7da23a874dc68f68eea1fe3d44be"
format_stamp: "Formatted at 2026-09-27 12:41:02 on dist-test-slave-bnc1"
server_key: "2c5d8d03fac2e5d61e2da4c41ba8a5f5"
server_key_iv: "d2541055ead98b619e5bc46fce37ef3c"
server_key_version: "encryptionkey@0"
I20260927 12:41:02.191308 3930 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 12:41:02.193560 3945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:02.194767 3930 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 12:41:02.194893 3930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/minicluster-data/master-0/wal
uuid: "665a7da23a874dc68f68eea1fe3d44be"
format_stamp: "Formatted at 2026-09-27 12:41:02 on dist-test-slave-bnc1"
server_key: "2c5d8d03fac2e5d61e2da4c41ba8a5f5"
server_key_iv: "d2541055ead98b619e5bc46fce37ef3c"
server_key_version: "encryptionkey@0"
I20260927 12:41:02.194989 3930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:41:02.212545 3930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:41:02.215341 3930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:41:02.215540 3930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:41:02.222493 3930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39743
I20260927 12:41:02.222513 3997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39743 every 8 connection(s)
I20260927 12:41:02.223431 3930 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:41:02.226001 3998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:41:02.230697 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 3930
I20260927 12:41:02.230794 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/minicluster-data/master-0/wal/instance
I20260927 12:41:02.231065 392 external_mini_cluster.cc:1428] Setting key 0677a729d0e8cffc34078eee31828fdf
I20260927 12:41:02.231277 3998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 665a7da23a874dc68f68eea1fe3d44be: Bootstrap starting.
I20260927 12:41:02.233901 3998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 665a7da23a874dc68f68eea1fe3d44be: Neither blocks nor log segments found. Creating new log.
I20260927 12:41:02.234740 3998 log.cc:824] T 00000000000000000000000000000000 P 665a7da23a874dc68f68eea1fe3d44be: Log is configured to *not* fsync() on all Append() calls
I20260927 12:41:02.236645 3998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 665a7da23a874dc68f68eea1fe3d44be: No bootstrap required, opened a new log
Sep 27 12:41:02 dist-test-slave-bnc1 krb5kdc[3917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:41:02.240988 3998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 665a7da23a874dc68f68eea1fe3d44be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "665a7da23a874dc68f68eea1fe3d44be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39743 } }
I20260927 12:41:02.241251 3998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 665a7da23a874dc68f68eea1fe3d44be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:41:02.241307 3998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 665a7da23a874dc68f68eea1fe3d44be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 665a7da23a874dc68f68eea1fe3d44be, State: Initialized, Role: FOLLOWER
I20260927 12:41:02.241858 3998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 665a7da23a874dc68f68eea1fe3d44be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "665a7da23a874dc68f68eea1fe3d44be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39743 } }
I20260927 12:41:02.242025 3998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 665a7da23a874dc68f68eea1fe3d44be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:41:02.242098 3998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 665a7da23a874dc68f68eea1fe3d44be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:41:02.242194 3998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 665a7da23a874dc68f68eea1fe3d44be [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:41:02.243274 3998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 665a7da23a874dc68f68eea1fe3d44be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "665a7da23a874dc68f68eea1fe3d44be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39743 } }
I20260927 12:41:02.243623 3998 leader_election.cc:302] T 00000000000000000000000000000000 P 665a7da23a874dc68f68eea1fe3d44be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 665a7da23a874dc68f68eea1fe3d44be; no voters:
I20260927 12:41:02.243938 3998 leader_election.cc:288] T 00000000000000000000000000000000 P 665a7da23a874dc68f68eea1fe3d44be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:41:02.244041 4003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 665a7da23a874dc68f68eea1fe3d44be [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:41:02.244277 4003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 665a7da23a874dc68f68eea1fe3d44be [term 1 LEADER]: Becoming Leader. State: Replica: 665a7da23a874dc68f68eea1fe3d44be, State: Running, Role: LEADER
I20260927 12:41:02.244596 4003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 665a7da23a874dc68f68eea1fe3d44be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "665a7da23a874dc68f68eea1fe3d44be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39743 } }
I20260927 12:41:02.245150 3998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 665a7da23a874dc68f68eea1fe3d44be [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:41:02.246251 4001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:02.232419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50746 (local address 127.0.98.62:39743)
0927 12:41:02.232815 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:02.232832 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:02.232872 (+ 40us) server_negotiation.cc:407] Connection header received
0927 12:41:02.233669 (+ 797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:02.233691 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:02.234100 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:02.234456 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:02.235199 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:02.236228 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:02.237359 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:02.237930 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:02.238045 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:02.240706 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:02.240711 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:02.240725 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:41:02.240753 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:02.242742 (+ 1989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:02.243140 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:02.243148 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:02.243165 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:02.243304 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:02.243626 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:02.243630 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:02.243635 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:02.244124 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:02.244832 (+ 708us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:02.244971 (+ 139us) server_negotiation.cc:299] Negotiation successful
0927 12:41:02.245263 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":101,"threads_started":1}
I20260927 12:41:02.246305 4005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 665a7da23a874dc68f68eea1fe3d44be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 665a7da23a874dc68f68eea1fe3d44be. Latest consensus state: current_term: 1 leader_uuid: "665a7da23a874dc68f68eea1fe3d44be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "665a7da23a874dc68f68eea1fe3d44be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39743 } } }
I20260927 12:41:02.246452 4005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 665a7da23a874dc68f68eea1fe3d44be [sys.catalog]: This master's current role is: LEADER
I20260927 12:41:02.246740 4004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 665a7da23a874dc68f68eea1fe3d44be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "665a7da23a874dc68f68eea1fe3d44be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "665a7da23a874dc68f68eea1fe3d44be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39743 } } }
I20260927 12:41:02.246794 4012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:41:02.246820 4004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 665a7da23a874dc68f68eea1fe3d44be [sys.catalog]: This master's current role is: LEADER
I20260927 12:41:02.249668 4012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:41:02.254130 4012 catalog_manager.cc:1347] Generated new cluster ID: 5b2ba73706df424099a98d7ee13393f6
I20260927 12:41:02.254187 4012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:41:02.271543 4012 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:41:02.272464 4012 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:41:02.276973 4012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 665a7da23a874dc68f68eea1fe3d44be: Generated new TSK 0
I20260927 12:41:02.277746 4012 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 12:41:02.311031 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39743
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:41:02.421491 4024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:41:02.421823 4024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:41:02.421927 4024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:41:02.425948 4024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:41:02.426033 4024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:41:02.426123 4024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:41:02.431223 4024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:41:02.432241 4024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:41:02.433261 4024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:41:02.442682 4032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:02.442683 4029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:02.442683 4030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:02.443066 4024 server_base.cc:1034] running on GCE node
I20260927 12:41:02.443440 4024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:41:02.444074 4024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:41:02.445222 4024 hybrid_clock.cc:648] HybridClock initialized: now 1790512862445187 us; error 44 us; skew 500 ppm
Sep 27 12:41:02 dist-test-slave-bnc1 krb5kdc[3917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512862, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:41:02.448359 4024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 12:41:02.449287 4024 webserver.cc:460] Webserver started at http://127.0.98.1:39291/ using document root <none> and password file <none>
I20260927 12:41:02.449922 4024 fs_manager.cc:359] Metadata directory not provided
I20260927 12:41:02.450018 4024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:41:02.450246 4024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:41:02.452310 4024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/minicluster-data/ts-0/data/instance:
uuid: "98a47029579940638e7778ca5686730d"
format_stamp: "Formatted at 2026-09-27 12:41:02 on dist-test-slave-bnc1"
server_key: "2404a5b24ad2a585b12f038e17d04938"
server_key_iv: "8706853a0c62e33912fea91b65610072"
server_key_version: "encryptionkey@0"
I20260927 12:41:02.452905 4024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/minicluster-data/ts-0/wal/instance:
uuid: "98a47029579940638e7778ca5686730d"
format_stamp: "Formatted at 2026-09-27 12:41:02 on dist-test-slave-bnc1"
server_key: "2404a5b24ad2a585b12f038e17d04938"
server_key_iv: "8706853a0c62e33912fea91b65610072"
server_key_version: "encryptionkey@0"
I20260927 12:41:02.456753 4024 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 12:41:02.459287 4039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:02.460510 4024 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:41:02.460649 4024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/minicluster-data/ts-0/wal
uuid: "98a47029579940638e7778ca5686730d"
format_stamp: "Formatted at 2026-09-27 12:41:02 on dist-test-slave-bnc1"
server_key: "2404a5b24ad2a585b12f038e17d04938"
server_key_iv: "8706853a0c62e33912fea91b65610072"
server_key_version: "encryptionkey@0"
I20260927 12:41:02.460790 4024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:41:02.484565 4024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:41:02.487696 4024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:41:02.487919 4024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:41:02.488515 4024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:41:02.489367 4024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:41:02.489442 4024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:02.489514 4024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:41:02.489557 4024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:02.498551 4024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36039
I20260927 12:41:02.498595 4152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36039 every 8 connection(s)
I20260927 12:41:02.499505 4024 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:41:02.507676 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 4024
I20260927 12:41:02.507814 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/minicluster-data/ts-0/wal/instance
I20260927 12:41:02.508107 392 external_mini_cluster.cc:1428] Setting key 0e2e8f9860f88faf9b0529a43dfa6312
Sep 27 12:41:02 dist-test-slave-bnc1 krb5kdc[3917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512862, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:41:02.513302 4001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:02.501276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55603 (local address 127.0.98.62:39743)
0927 12:41:02.501409 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:02.501420 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:02.502239 (+ 819us) server_negotiation.cc:407] Connection header received
0927 12:41:02.503660 (+ 1421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:02.503663 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:02.503723 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:02.503834 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:02.504973 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:02.505516 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:02.506669 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:02.506893 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:02.506970 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:02.509541 (+ 2571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:02.509545 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:02.509548 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:41:02.509600 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:02.511035 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:02.511622 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:02.511624 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:02.511626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:02.511693 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:02.512077 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:02.512079 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:02.512080 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:02.512358 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:02.512463 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:02.512905 (+ 442us) server_negotiation.cc:299] Negotiation successful
0927 12:41:02.513045 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 12:41:02.513901 4155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:02.501641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39743 (local address 127.0.98.1:55603)
0927 12:41:02.502082 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:02.502127 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:02.503447 (+ 1320us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:02.504006 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:02.504013 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:02.504355 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:02.504823 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:02.504835 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:02.505703 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:02.505706 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:02.506569 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:02.506578 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:02.507015 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:02.507017 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:02.507147 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:02.507789 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:02.507811 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:02.509397 (+ 1586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:41:02.511149 (+ 1752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:02.511157 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:02.511173 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:02.511526 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:02.511780 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:02.511783 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:02.511786 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:02.511980 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:02.512461 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:41:02.512466 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:41:02.512674 (+ 208us) client_negotiation.cc:769] Sending connection context
0927 12:41:02.512823 (+ 149us) client_negotiation.cc:240] Negotiation successful
0927 12:41:02.513058 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":109,"threads_started":1}
I20260927 12:41:02.515152 4153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39743
I20260927 12:41:02.515403 4153 heartbeater.cc:464] Registering TS with master...
I20260927 12:41:02.515864 4153 heartbeater.cc:511] Master 127.0.98.62:39743 requested a full tablet report, sending...
I20260927 12:41:02.517292 3962 ts_manager.cc:194] Registered new tserver with Master: 98a47029579940638e7778ca5686730d (127.0.98.1:36039)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 12:41:02.518477 3962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55603
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 12:41:02.536000 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39743
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:41:02.647717 4158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:41:02.648008 4158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:41:02.648079 4158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:41:02.652055 4158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:41:02.652140 4158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:41:02.652227 4158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:41:02.657385 4158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:41:02.658473 4158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:41:02.659492 4158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:41:02.669332 4163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:02.669332 4164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:02.669554 4158 server_base.cc:1034] running on GCE node
W20260927 12:41:02.669348 4166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:02.670024 4158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:41:02.670655 4158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:41:02.671835 4158 hybrid_clock.cc:648] HybridClock initialized: now 1790512862671824 us; error 28 us; skew 500 ppm
Sep 27 12:41:02 dist-test-slave-bnc1 krb5kdc[3917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512862, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:41:02.674644 4158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 12:41:02.675531 4158 webserver.cc:460] Webserver started at http://127.0.98.2:35925/ using document root <none> and password file <none>
I20260927 12:41:02.676028 4158 fs_manager.cc:359] Metadata directory not provided
I20260927 12:41:02.676084 4158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:41:02.676240 4158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:41:02.678225 4158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/minicluster-data/ts-1/data/instance:
uuid: "8702333453e240a394d98f528659926b"
format_stamp: "Formatted at 2026-09-27 12:41:02 on dist-test-slave-bnc1"
server_key: "e9d81ea5682986178771e1633a0623ee"
server_key_iv: "ccc6e3fad4431852d41cb09c37cd92d6"
server_key_version: "encryptionkey@0"
I20260927 12:41:02.678717 4158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8702333453e240a394d98f528659926b"
format_stamp: "Formatted at 2026-09-27 12:41:02 on dist-test-slave-bnc1"
server_key: "e9d81ea5682986178771e1633a0623ee"
server_key_iv: "ccc6e3fad4431852d41cb09c37cd92d6"
server_key_version: "encryptionkey@0"
I20260927 12:41:02.682261 4158 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 12:41:02.684492 4173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:02.685706 4158 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 12:41:02.685812 4158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/minicluster-data/ts-1/wal
uuid: "8702333453e240a394d98f528659926b"
format_stamp: "Formatted at 2026-09-27 12:41:02 on dist-test-slave-bnc1"
server_key: "e9d81ea5682986178771e1633a0623ee"
server_key_iv: "ccc6e3fad4431852d41cb09c37cd92d6"
server_key_version: "encryptionkey@0"
I20260927 12:41:02.685911 4158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:41:02.702287 4158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:41:02.705019 4158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:41:02.705216 4158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:41:02.705838 4158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:41:02.706620 4158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:41:02.706665 4158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:02.706730 4158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:41:02.706769 4158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:02.715232 4158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43287
I20260927 12:41:02.715256 4286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43287 every 8 connection(s)
I20260927 12:41:02.716027 4158 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:41:02.722142 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 4158
I20260927 12:41:02.722285 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/minicluster-data/ts-1/wal/instance
I20260927 12:41:02.722590 392 external_mini_cluster.cc:1428] Setting key c3f2348f4203ac3dad5bcb49102c09c4
Sep 27 12:41:02 dist-test-slave-bnc1 krb5kdc[3917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512862, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:41:02.730767 4001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:02.717934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60675 (local address 127.0.98.62:39743)
0927 12:41:02.718063 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:02.718067 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:02.718895 (+ 828us) server_negotiation.cc:407] Connection header received
0927 12:41:02.720377 (+ 1482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:02.720380 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:02.720434 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:02.720525 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:02.721632 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:02.722166 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:02.723672 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:02.724012 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:02.724096 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:02.727071 (+ 2975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:02.727074 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:02.727076 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:41:02.727098 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:02.728487 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:02.729109 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:02.729112 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:02.729113 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:02.729181 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:02.729634 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:02.729636 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:02.729637 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:02.729803 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:02.729877 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:02.730526 (+ 649us) server_negotiation.cc:299] Negotiation successful
0927 12:41:02.730618 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 12:41:02.731951 4289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:02.718328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39743 (local address 127.0.98.2:60675)
0927 12:41:02.718760 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:02.718799 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:02.720181 (+ 1382us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:02.720705 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:02.720712 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:02.721035 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:02.721479 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:02.721491 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:02.722349 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:02.722353 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:02.723560 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:02.723569 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:02.724157 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:02.724162 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:02.724377 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:02.724967 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:02.724991 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:02.726864 (+ 1873us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:41:02.728602 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:02.728609 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:02.728626 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:02.729005 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:02.729271 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:02.729276 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:02.729279 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:02.729488 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:02.729911 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:41:02.729919 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:41:02.730255 (+ 336us) client_negotiation.cc:769] Sending connection context
0927 12:41:02.730483 (+ 228us) client_negotiation.cc:240] Negotiation successful
0927 12:41:02.730873 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":120,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 12:41:02.733125 4287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39743
I20260927 12:41:02.733335 4287 heartbeater.cc:464] Registering TS with master...
I20260927 12:41:02.733883 4287 heartbeater.cc:511] Master 127.0.98.62:39743 requested a full tablet report, sending...
I20260927 12:41:02.734877 3962 ts_manager.cc:194] Registered new tserver with Master: 8702333453e240a394d98f528659926b (127.0.98.2:43287)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 12:41:02.735713 3962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60675
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 12:41:02.750860 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39743
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:41:02.860443 4292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:41:02.860766 4292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:41:02.860854 4292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:41:02.865061 4292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:41:02.865145 4292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:41:02.865269 4292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:41:02.870616 4292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:41:02.871624 4292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:41:02.872617 4292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:41:02.882635 4298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:02.882635 4300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:02.882735 4292 server_base.cc:1034] running on GCE node
W20260927 12:41:02.882671 4297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:02.883376 4292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:41:02.884020 4292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:41:02.885182 4292 hybrid_clock.cc:648] HybridClock initialized: now 1790512862885165 us; error 28 us; skew 500 ppm
Sep 27 12:41:02 dist-test-slave-bnc1 krb5kdc[3917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512862, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:41:02.888019 4292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 12:41:02.888914 4292 webserver.cc:460] Webserver started at http://127.0.98.3:35113/ using document root <none> and password file <none>
I20260927 12:41:02.889528 4292 fs_manager.cc:359] Metadata directory not provided
I20260927 12:41:02.889683 4292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:41:02.889897 4292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:41:02.891880 4292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/minicluster-data/ts-2/data/instance:
uuid: "93cacab2f00f4cc88317589028955e20"
format_stamp: "Formatted at 2026-09-27 12:41:02 on dist-test-slave-bnc1"
server_key: "27c5ac11863d404354f37c35fad146fe"
server_key_iv: "950c21f63dd0a755ae9aadded907715a"
server_key_version: "encryptionkey@0"
I20260927 12:41:02.892388 4292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/minicluster-data/ts-2/wal/instance:
uuid: "93cacab2f00f4cc88317589028955e20"
format_stamp: "Formatted at 2026-09-27 12:41:02 on dist-test-slave-bnc1"
server_key: "27c5ac11863d404354f37c35fad146fe"
server_key_iv: "950c21f63dd0a755ae9aadded907715a"
server_key_version: "encryptionkey@0"
I20260927 12:41:02.896059 4292 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 12:41:02.898591 4307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:02.899988 4292 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 12:41:02.900113 4292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/minicluster-data/ts-2/wal
uuid: "93cacab2f00f4cc88317589028955e20"
format_stamp: "Formatted at 2026-09-27 12:41:02 on dist-test-slave-bnc1"
server_key: "27c5ac11863d404354f37c35fad146fe"
server_key_iv: "950c21f63dd0a755ae9aadded907715a"
server_key_version: "encryptionkey@0"
I20260927 12:41:02.900221 4292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:41:02.929522 4292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:41:02.933151 4292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:41:02.933367 4292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:41:02.934118 4292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:41:02.934950 4292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:41:02.934998 4292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:02.935070 4292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:41:02.935113 4292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:02.944077 4292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33315
I20260927 12:41:02.944098 4420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33315 every 8 connection(s)
I20260927 12:41:02.945089 4292 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:41:02.946954 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 4292
I20260927 12:41:02.947088 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790512840836255-392-0/minicluster-data/ts-2/wal/instance
I20260927 12:41:02.947410 392 external_mini_cluster.cc:1428] Setting key 0def863bac176a697ed9561fd0fb6cd4
Sep 27 12:41:02 dist-test-slave-bnc1 krb5kdc[3917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512862, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:41:02.959254 4001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:02.947169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38521 (local address 127.0.98.62:39743)
0927 12:41:02.947328 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:02.947333 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:02.948455 (+ 1122us) server_negotiation.cc:407] Connection header received
0927 12:41:02.950104 (+ 1649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:02.950107 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:02.950151 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:02.950220 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:02.951425 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:02.951942 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:02.953042 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:02.953261 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:02.953337 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:02.955985 (+ 2648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:02.955988 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:02.955990 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:41:02.956014 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:02.957351 (+ 1337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:02.957918 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:02.957920 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:02.957921 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:02.957992 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:02.958305 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:02.958307 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:02.958308 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:02.958471 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:02.958543 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:02.959020 (+ 477us) server_negotiation.cc:299] Negotiation successful
0927 12:41:02.959129 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 12:41:02.960120 4423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:02.947717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39743 (local address 127.0.98.3:38521)
0927 12:41:02.948273 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:02.948352 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:02.949902 (+ 1550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:02.950381 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:02.950388 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:02.950747 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:02.951289 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:02.951302 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:02.952048 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:02.952051 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:02.952941 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:02.952949 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:02.953422 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:02.953425 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:02.953649 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:02.954352 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:02.954369 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:02.955823 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:41:02.957469 (+ 1646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:02.957474 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:02.957486 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:02.957832 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:02.958085 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:02.958087 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:02.958089 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:02.958222 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:02.958558 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:41:02.958562 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:41:02.958800 (+ 238us) client_negotiation.cc:769] Sending connection context
0927 12:41:02.958948 (+ 148us) client_negotiation.cc:240] Negotiation successful
0927 12:41:02.959209 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":146,"threads_started":1}
I20260927 12:41:02.961294 4421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39743
I20260927 12:41:02.961529 4421 heartbeater.cc:464] Registering TS with master...
I20260927 12:41:02.962045 4421 heartbeater.cc:511] Master 127.0.98.62:39743 requested a full tablet report, sending...
I20260927 12:41:02.962993 3962 ts_manager.cc:194] Registered new tserver with Master: 93cacab2f00f4cc88317589028955e20 (127.0.98.3:33315)
I20260927 12:41:02.963579 3962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38521
I20260927 12:41:02.973299 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 12:41:02 dist-test-slave-bnc1 krb5kdc[3917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 12:41:02 dist-test-slave-bnc1 krb5kdc[3917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 12:41:02.998734 4428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:02.989598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45246 (local address 127.0.98.1:36039)
0927 12:41:02.989966 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:02.989980 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:02.990009 (+ 29us) server_negotiation.cc:407] Connection header received
0927 12:41:02.990060 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:02.990065 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:02.990257 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:02.990384 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:02.990992 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:02.991745 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:02.992527 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:02.992882 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:02.992963 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:02.995276 (+ 2313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:02.995284 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:02.995297 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:41:02.995320 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:02.997016 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:02.997375 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:02.997379 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:02.997381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:02.997456 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:02.997782 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:02.997785 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:02.997786 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:02.998021 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:02.998187 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:02.998345 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 12:41:02.998517 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":88,"threads_started":1}
Sep 27 12:41:03 dist-test-slave-bnc1 krb5kdc[3917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:41:03.009783 4001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:03.001747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50758 (local address 127.0.98.62:39743)
0927 12:41:03.001927 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:03.001931 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:03.001975 (+ 44us) server_negotiation.cc:407] Connection header received
0927 12:41:03.002082 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:03.002085 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:03.002122 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:03.002190 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:03.002771 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:03.003277 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:03.004048 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:03.004311 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:03.004400 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:03.006751 (+ 2351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:03.006754 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:03.006756 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:41:03.006779 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:03.008196 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:03.008607 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:03.008610 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:03.008611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:03.008682 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:03.008904 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:03.008906 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:03.008907 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:03.009115 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:03.009219 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:03.009429 (+ 210us) server_negotiation.cc:299] Negotiation successful
0927 12:41:03.009562 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
W20260927 12:41:03.010329 3962 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50758
I20260927 12:41:03.010959 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 4024
I20260927 12:41:03.016881 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 4158
I20260927 12:41:03.022231 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 4292
I20260927 12:41:03.027572 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 3930
2026-09-27T12:41:03Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2139 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:41:03 dist-test-slave-bnc1 krb5kdc[4433](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:41:03 dist-test-slave-bnc1 krb5kdc[4433](info): set up 2 sockets
Sep 27 12:41:03 dist-test-slave-bnc1 krb5kdc[4433](info): commencing operation
krb5kdc: starting...
W20260927 12:41:04.073767 392 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/krb5kdc/test-user.keytab.
Sep 27 12:41:04 dist-test-slave-bnc1 krb5kdc[4433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:41:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:41:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:41:04.198948 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32991
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32991
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:41:04.307154 4446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:41:04.307458 4446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:41:04.307499 4446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:41:04.311542 4446 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:41:04.311604 4446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:41:04.311623 4446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:41:04.311640 4446 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:41:04.311655 4446 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:41:04.317057 4446 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32991
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4446
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:41:04.318117 4446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:41:04.319151 4446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:41:04.327492 4451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:04.327477 4454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:04.327661 4446 server_base.cc:1034] running on GCE node
W20260927 12:41:04.327507 4452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:04.328361 4446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:41:04.329470 4446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:41:04.330650 4446 hybrid_clock.cc:648] HybridClock initialized: now 1790512864330626 us; error 33 us; skew 500 ppm
Sep 27 12:41:04 dist-test-slave-bnc1 krb5kdc[4433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512864, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:41:04.333346 4446 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:41:04.334095 4446 fs_manager.cc:359] Metadata directory not provided
I20260927 12:41:04.334200 4446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:41:04.334426 4446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:41:04.336432 4446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/minicluster-data/master-0/data/instance:
uuid: "8c56efcb986546e09a7c3265f21e86a1"
format_stamp: "Formatted at 2026-09-27 12:41:04 on dist-test-slave-bnc1"
server_key: "4bb929f21e67e12490dc699d332771cf"
server_key_iv: "6d221696ea5cf42f0f794e8c16878467"
server_key_version: "encryptionkey@0"
I20260927 12:41:04.336994 4446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/minicluster-data/master-0/wal/instance:
uuid: "8c56efcb986546e09a7c3265f21e86a1"
format_stamp: "Formatted at 2026-09-27 12:41:04 on dist-test-slave-bnc1"
server_key: "4bb929f21e67e12490dc699d332771cf"
server_key_iv: "6d221696ea5cf42f0f794e8c16878467"
server_key_version: "encryptionkey@0"
I20260927 12:41:04.340613 4446 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.004s
I20260927 12:41:04.343281 4460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:04.344635 4446 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 12:41:04.344805 4446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/minicluster-data/master-0/wal
uuid: "8c56efcb986546e09a7c3265f21e86a1"
format_stamp: "Formatted at 2026-09-27 12:41:04 on dist-test-slave-bnc1"
server_key: "4bb929f21e67e12490dc699d332771cf"
server_key_iv: "6d221696ea5cf42f0f794e8c16878467"
server_key_version: "encryptionkey@0"
I20260927 12:41:04.344941 4446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:41:04.364347 4446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:41:04.368005 4446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:41:04.368252 4446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:41:04.375571 4446 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32991
I20260927 12:41:04.375836 4512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32991 every 8 connection(s)
I20260927 12:41:04.376284 4446 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:41:04.379076 4513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:41:04.384675 4513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c56efcb986546e09a7c3265f21e86a1: Bootstrap starting.
I20260927 12:41:04.384895 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 4446
I20260927 12:41:04.385056 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/minicluster-data/master-0/wal/instance
I20260927 12:41:04.385311 392 external_mini_cluster.cc:1428] Setting key 619303d8344dcb0ebaf643b7190d5be5
I20260927 12:41:04.387612 4513 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c56efcb986546e09a7c3265f21e86a1: Neither blocks nor log segments found. Creating new log.
I20260927 12:41:04.388535 4513 log.cc:824] T 00000000000000000000000000000000 P 8c56efcb986546e09a7c3265f21e86a1: Log is configured to *not* fsync() on all Append() calls
I20260927 12:41:04.390872 4513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c56efcb986546e09a7c3265f21e86a1: No bootstrap required, opened a new log
Sep 27 12:41:04 dist-test-slave-bnc1 krb5kdc[4433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:41:04.395083 4513 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c56efcb986546e09a7c3265f21e86a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c56efcb986546e09a7c3265f21e86a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32991 } }
I20260927 12:41:04.395392 4513 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c56efcb986546e09a7c3265f21e86a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:41:04.395483 4513 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c56efcb986546e09a7c3265f21e86a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c56efcb986546e09a7c3265f21e86a1, State: Initialized, Role: FOLLOWER
I20260927 12:41:04.396034 4513 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c56efcb986546e09a7c3265f21e86a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c56efcb986546e09a7c3265f21e86a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32991 } }
I20260927 12:41:04.396198 4513 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c56efcb986546e09a7c3265f21e86a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:41:04.396274 4513 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c56efcb986546e09a7c3265f21e86a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:41:04.396380 4513 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c56efcb986546e09a7c3265f21e86a1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:41:04.397665 4513 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c56efcb986546e09a7c3265f21e86a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c56efcb986546e09a7c3265f21e86a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32991 } }
I20260927 12:41:04.398061 4513 leader_election.cc:302] T 00000000000000000000000000000000 P 8c56efcb986546e09a7c3265f21e86a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c56efcb986546e09a7c3265f21e86a1; no voters:
I20260927 12:41:04.398495 4513 leader_election.cc:288] T 00000000000000000000000000000000 P 8c56efcb986546e09a7c3265f21e86a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:41:04.398574 4518 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c56efcb986546e09a7c3265f21e86a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:41:04.398826 4518 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c56efcb986546e09a7c3265f21e86a1 [term 1 LEADER]: Becoming Leader. State: Replica: 8c56efcb986546e09a7c3265f21e86a1, State: Running, Role: LEADER
I20260927 12:41:04.399183 4518 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c56efcb986546e09a7c3265f21e86a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c56efcb986546e09a7c3265f21e86a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32991 } }
I20260927 12:41:04.399935 4513 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c56efcb986546e09a7c3265f21e86a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:41:04.401078 4519 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c56efcb986546e09a7c3265f21e86a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c56efcb986546e09a7c3265f21e86a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c56efcb986546e09a7c3265f21e86a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32991 } } }
I20260927 12:41:04.401293 4519 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c56efcb986546e09a7c3265f21e86a1 [sys.catalog]: This master's current role is: LEADER
I20260927 12:41:04.401490 4516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:04.386783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42348 (local address 127.0.98.62:32991)
0927 12:41:04.387207 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:04.387225 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:04.387267 (+ 42us) server_negotiation.cc:407] Connection header received
0927 12:41:04.388177 (+ 910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:04.388201 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:04.388639 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:04.389038 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:04.389912 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:04.391322 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:04.392158 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:04.392884 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:04.393042 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:04.395765 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:04.395771 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:04.395788 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:41:04.395820 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:04.397978 (+ 2158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:04.398532 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:04.398538 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:04.398546 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:04.398733 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:04.399070 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:04.399073 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:04.399074 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:04.399558 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:04.399787 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:04.400098 (+ 311us) server_negotiation.cc:299] Negotiation successful
0927 12:41:04.400435 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":116,"threads_started":1}
I20260927 12:41:04.401728 4519 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c56efcb986546e09a7c3265f21e86a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c56efcb986546e09a7c3265f21e86a1. Latest consensus state: current_term: 1 leader_uuid: "8c56efcb986546e09a7c3265f21e86a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c56efcb986546e09a7c3265f21e86a1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32991 } } }
I20260927 12:41:04.401851 4519 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c56efcb986546e09a7c3265f21e86a1 [sys.catalog]: This master's current role is: LEADER
W20260927 12:41:04.405063 4533 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8c56efcb986546e09a7c3265f21e86a1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 12:41:04.405143 4533 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 12:41:04.405203 4527 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:41:04.406147 4527 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:41:04.412197 4527 catalog_manager.cc:1347] Generated new cluster ID: 008c6c8b870e4f6ebcd98e35dc5500ec
I20260927 12:41:04.412294 4527 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:41:04.429167 4527 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:41:04.430510 4527 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:41:04.442312 4527 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c56efcb986546e09a7c3265f21e86a1: Generated new TSK 0
I20260927 12:41:04.443091 4527 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 12:41:04.477938 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32991
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:41:04.590870 4539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:41:04.591146 4539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:41:04.591192 4539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:41:04.595327 4539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:41:04.595389 4539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:41:04.595450 4539 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:41:04.600977 4539 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32991
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4539
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:41:04.602195 4539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:41:04.603200 4539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:41:04.614723 4544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:04.614837 4547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:04.614723 4545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:04.615926 4539 server_base.cc:1034] running on GCE node
I20260927 12:41:04.616503 4539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:41:04.617201 4539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:41:04.618374 4539 hybrid_clock.cc:648] HybridClock initialized: now 1790512864618328 us; error 60 us; skew 500 ppm
Sep 27 12:41:04 dist-test-slave-bnc1 krb5kdc[4433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512864, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:41:04.621349 4539 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 12:41:04.622098 4539 fs_manager.cc:359] Metadata directory not provided
I20260927 12:41:04.622161 4539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:41:04.622323 4539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:41:04.624323 4539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/minicluster-data/ts-0/data/instance:
uuid: "fd02a2dcafa748e192aff8c7b76fd85b"
format_stamp: "Formatted at 2026-09-27 12:41:04 on dist-test-slave-bnc1"
server_key: "1f73f854df8aa704e53df21ac0ba42de"
server_key_iv: "1657c7a038e3dc1e1bd46bb0071cd442"
server_key_version: "encryptionkey@0"
I20260927 12:41:04.624847 4539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/minicluster-data/ts-0/wal/instance:
uuid: "fd02a2dcafa748e192aff8c7b76fd85b"
format_stamp: "Formatted at 2026-09-27 12:41:04 on dist-test-slave-bnc1"
server_key: "1f73f854df8aa704e53df21ac0ba42de"
server_key_iv: "1657c7a038e3dc1e1bd46bb0071cd442"
server_key_version: "encryptionkey@0"
I20260927 12:41:04.628468 4539 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 12:41:04.630863 4553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:04.632026 4539 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 12:41:04.632151 4539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/minicluster-data/ts-0/wal
uuid: "fd02a2dcafa748e192aff8c7b76fd85b"
format_stamp: "Formatted at 2026-09-27 12:41:04 on dist-test-slave-bnc1"
server_key: "1f73f854df8aa704e53df21ac0ba42de"
server_key_iv: "1657c7a038e3dc1e1bd46bb0071cd442"
server_key_version: "encryptionkey@0"
I20260927 12:41:04.632249 4539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:41:04.660535 4539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:41:04.663815 4539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:41:04.664031 4539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:41:04.664649 4539 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:41:04.665669 4539 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:41:04.665716 4539 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:04.665781 4539 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:41:04.665812 4539 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:04.674721 4539 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41985
I20260927 12:41:04.674770 4666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41985 every 8 connection(s)
I20260927 12:41:04.675333 4539 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
Sep 27 12:41:04 dist-test-slave-bnc1 krb5kdc[4433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512864, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:41:04.684986 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 4539
I20260927 12:41:04.685120 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/minicluster-data/ts-0/wal/instance
I20260927 12:41:04.685429 392 external_mini_cluster.cc:1428] Setting key 3559d27ef5a08d2ecf17d830ea9068f4
I20260927 12:41:04.689276 4516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:04.677033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37057 (local address 127.0.98.62:32991)
0927 12:41:04.677199 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:04.677203 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:04.678064 (+ 861us) server_negotiation.cc:407] Connection header received
0927 12:41:04.679447 (+ 1383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:04.679450 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:04.679506 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:04.679595 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:04.680763 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:04.681314 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:04.682536 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:04.682780 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:04.682906 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:04.685606 (+ 2700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:04.685611 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:04.685614 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:41:04.685645 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:04.687254 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:04.687756 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:04.687761 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:04.687763 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:04.687843 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:04.688187 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:04.688192 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:04.688193 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:04.688381 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:04.688488 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:04.688996 (+ 508us) server_negotiation.cc:299] Negotiation successful
0927 12:41:04.689118 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 12:41:04.690186 4669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:04.677413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32991 (local address 127.0.98.1:37057)
0927 12:41:04.677916 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:04.677959 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:04.679216 (+ 1257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:04.679784 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:04.679791 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:04.680138 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:04.680615 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:04.680628 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:04.681465 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:04.681468 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:04.682410 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:04.682419 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:04.682943 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:04.682947 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:04.683116 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:04.683732 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:04.683748 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:04.685402 (+ 1654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:41:04.687368 (+ 1966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:04.687376 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:04.687392 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:04.687649 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:04.687938 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:04.687942 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:04.687944 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:04.688091 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:04.688486 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:41:04.688493 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:41:04.688746 (+ 253us) client_negotiation.cc:769] Sending connection context
0927 12:41:04.688946 (+ 200us) client_negotiation.cc:240] Negotiation successful
0927 12:41:04.689276 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":165,"threads_started":1}
I20260927 12:41:04.691558 4667 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32991
I20260927 12:41:04.691797 4667 heartbeater.cc:464] Registering TS with master...
I20260927 12:41:04.692235 4667 heartbeater.cc:511] Master 127.0.98.62:32991 requested a full tablet report, sending...
I20260927 12:41:04.693748 4477 ts_manager.cc:194] Registered new tserver with Master: fd02a2dcafa748e192aff8c7b76fd85b (127.0.98.1:41985)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 12:41:04.694880 4477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37057
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 12:41:04.712558 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32991
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:41:04.821456 4672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:41:04.821820 4672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:41:04.821899 4672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:41:04.825911 4672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:41:04.825995 4672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:41:04.826089 4672 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:41:04.831257 4672 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32991
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4672
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:41:04.832276 4672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:41:04.833280 4672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:41:04.842631 4677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:04.842692 4680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:04.842631 4678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:04.843003 4672 server_base.cc:1034] running on GCE node
I20260927 12:41:04.843495 4672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:41:04.844174 4672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:41:04.845353 4672 hybrid_clock.cc:648] HybridClock initialized: now 1790512864845318 us; error 51 us; skew 500 ppm
Sep 27 12:41:04 dist-test-slave-bnc1 krb5kdc[4433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512864, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:41:04.848284 4672 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 12:41:04.849097 4672 fs_manager.cc:359] Metadata directory not provided
I20260927 12:41:04.849159 4672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:41:04.849352 4672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:41:04.851411 4672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/minicluster-data/ts-1/data/instance:
uuid: "3e11aaa147694132b213c760286d38e2"
format_stamp: "Formatted at 2026-09-27 12:41:04 on dist-test-slave-bnc1"
server_key: "929a7b737c5745d60bab4f7bb40a58e3"
server_key_iv: "a066daa0616b66a6aee07a95d8d07d42"
server_key_version: "encryptionkey@0"
I20260927 12:41:04.851912 4672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3e11aaa147694132b213c760286d38e2"
format_stamp: "Formatted at 2026-09-27 12:41:04 on dist-test-slave-bnc1"
server_key: "929a7b737c5745d60bab4f7bb40a58e3"
server_key_iv: "a066daa0616b66a6aee07a95d8d07d42"
server_key_version: "encryptionkey@0"
I20260927 12:41:04.855994 4672 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 12:41:04.858996 4686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:04.860191 4672 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 12:41:04.860327 4672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/minicluster-data/ts-1/wal
uuid: "3e11aaa147694132b213c760286d38e2"
format_stamp: "Formatted at 2026-09-27 12:41:04 on dist-test-slave-bnc1"
server_key: "929a7b737c5745d60bab4f7bb40a58e3"
server_key_iv: "a066daa0616b66a6aee07a95d8d07d42"
server_key_version: "encryptionkey@0"
I20260927 12:41:04.860436 4672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:41:04.882030 4672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:41:04.884968 4672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:41:04.885175 4672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:41:04.885793 4672 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:41:04.886619 4672 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:41:04.886693 4672 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:04.886765 4672 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:41:04.886807 4672 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:04.895920 4672 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39517
I20260927 12:41:04.895937 4799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39517 every 8 connection(s)
I20260927 12:41:04.896536 4672 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:41:04.898877 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 4672
I20260927 12:41:04.899024 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/minicluster-data/ts-1/wal/instance
I20260927 12:41:04.899364 392 external_mini_cluster.cc:1428] Setting key b8b05159567d6ffc218165519e2072c9
Sep 27 12:41:04 dist-test-slave-bnc1 krb5kdc[4433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512864, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 12:41:04.911832 4516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:04.898293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50695 (local address 127.0.98.62:32991)
0927 12:41:04.898432 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:04.898441 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:04.899463 (+ 1022us) server_negotiation.cc:407] Connection header received
0927 12:41:04.901293 (+ 1830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:04.901297 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:04.901348 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:04.901439 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:04.903162 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:04.904030 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:04.905171 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:04.905396 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:04.905516 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:04.908000 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:04.908003 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:04.908005 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:41:04.908029 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:04.909480 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:04.910085 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:04.910091 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:04.910093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:04.910173 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:04.910527 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:04.910530 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:04.910532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:04.910790 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:04.910901 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:04.911475 (+ 574us) server_negotiation.cc:299] Negotiation successful
0927 12:41:04.911617 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 12:41:04.912845 4802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:04.898759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32991 (local address 127.0.98.2:50695)
0927 12:41:04.899298 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:04.899340 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:04.901064 (+ 1724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:04.901707 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:04.901721 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:04.902275 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:04.902988 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:04.903005 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:04.904155 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:04.904157 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:04.905042 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:04.905051 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:04.905554 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:04.905558 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:04.905754 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:04.906372 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:04.906390 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:04.907813 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:41:04.909620 (+ 1807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:04.909627 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:04.909643 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:04.909974 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:04.910276 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:04.910279 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:04.910281 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:04.910427 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:04.910898 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:41:04.910904 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:41:04.911157 (+ 253us) client_negotiation.cc:769] Sending connection context
0927 12:41:04.911345 (+ 188us) client_negotiation.cc:240] Negotiation successful
0927 12:41:04.911677 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":138,"threads_started":1}
I20260927 12:41:04.914168 4800 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32991
I20260927 12:41:04.914431 4800 heartbeater.cc:464] Registering TS with master...
I20260927 12:41:04.914891 4800 heartbeater.cc:511] Master 127.0.98.62:32991 requested a full tablet report, sending...
I20260927 12:41:04.916074 4477 ts_manager.cc:194] Registered new tserver with Master: 3e11aaa147694132b213c760286d38e2 (127.0.98.2:39517)
I20260927 12:41:04.916658 4477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50695
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 12:41:04.929241 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32991
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:41:05.039440 4805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:41:05.039727 4805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:41:05.039832 4805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:41:05.044202 4805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:41:05.044291 4805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:41:05.044380 4805 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:41:05.049935 4805 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32991
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:41:05.051043 4805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:41:05.052079 4805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:41:05.061473 4810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:05.061559 4805 server_base.cc:1034] running on GCE node
W20260927 12:41:05.061513 4811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:05.061478 4813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:05.062219 4805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:41:05.062870 4805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:41:05.064054 4805 hybrid_clock.cc:648] HybridClock initialized: now 1790512865064026 us; error 42 us; skew 500 ppm
Sep 27 12:41:05 dist-test-slave-bnc1 krb5kdc[4433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512865, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:41:05.066978 4805 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 12:41:05.067627 4805 fs_manager.cc:359] Metadata directory not provided
I20260927 12:41:05.067687 4805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:41:05.067840 4805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:41:05.069844 4805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/minicluster-data/ts-2/data/instance:
uuid: "2c0272c693574844aea11c6e416121ed"
format_stamp: "Formatted at 2026-09-27 12:41:05 on dist-test-slave-bnc1"
server_key: "fb29a086dcf7b8ebe70e61fedac212ff"
server_key_iv: "c3662efaeb47949fe793e88ebbea011a"
server_key_version: "encryptionkey@0"
I20260927 12:41:05.070348 4805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2c0272c693574844aea11c6e416121ed"
format_stamp: "Formatted at 2026-09-27 12:41:05 on dist-test-slave-bnc1"
server_key: "fb29a086dcf7b8ebe70e61fedac212ff"
server_key_iv: "c3662efaeb47949fe793e88ebbea011a"
server_key_version: "encryptionkey@0"
I20260927 12:41:05.073841 4805 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 12:41:05.076145 4819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:05.077340 4805 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:41:05.077468 4805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/minicluster-data/ts-2/wal
uuid: "2c0272c693574844aea11c6e416121ed"
format_stamp: "Formatted at 2026-09-27 12:41:05 on dist-test-slave-bnc1"
server_key: "fb29a086dcf7b8ebe70e61fedac212ff"
server_key_iv: "c3662efaeb47949fe793e88ebbea011a"
server_key_version: "encryptionkey@0"
I20260927 12:41:05.077610 4805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:41:05.092669 4805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:41:05.095913 4805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:41:05.096117 4805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:41:05.096674 4805 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:41:05.097478 4805 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:41:05.097525 4805 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:05.097615 4805 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:41:05.097654 4805 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:05.106190 4805 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41727
I20260927 12:41:05.106241 4932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41727 every 8 connection(s)
I20260927 12:41:05.106752 4805 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:41:05.115307 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 4805
I20260927 12:41:05.115559 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/minicluster-data/ts-2/wal/instance
I20260927 12:41:05.115854 392 external_mini_cluster.cc:1428] Setting key d1038aacf6dd92c1cd244bd4f0e838d5
Sep 27 12:41:05 dist-test-slave-bnc1 krb5kdc[4433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512865, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:41:05.120405 4516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:05.108453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59649 (local address 127.0.98.62:32991)
0927 12:41:05.108610 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:05.108615 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:05.109538 (+ 923us) server_negotiation.cc:407] Connection header received
0927 12:41:05.110931 (+ 1393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:05.110934 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:05.110981 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:05.111060 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:05.112137 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:05.112656 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:05.113764 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:05.113972 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:05.114067 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:05.116575 (+ 2508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:05.116579 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:05.116581 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:41:05.116613 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:05.118383 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:05.118888 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:05.118892 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:05.118894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:05.118978 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:05.119324 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:05.119327 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:05.119329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:05.119512 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:05.119621 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:05.120119 (+ 498us) server_negotiation.cc:299] Negotiation successful
0927 12:41:05.120238 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 12:41:05.121217 4935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:05.108906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32991 (local address 127.0.98.3:59649)
0927 12:41:05.109377 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:05.109416 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:05.110714 (+ 1298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:05.111214 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:05.111221 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:05.111544 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:05.112012 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:05.112023 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:05.112755 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:05.112758 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:05.113638 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:05.113649 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:05.114109 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:05.114112 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:05.114241 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:05.114826 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:05.114840 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:05.116388 (+ 1548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:41:05.118491 (+ 2103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:05.118498 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:05.118511 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:05.118775 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:05.119081 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:05.119084 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:05.119086 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:05.119223 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:05.119624 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:41:05.119633 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:41:05.119879 (+ 246us) client_negotiation.cc:769] Sending connection context
0927 12:41:05.120069 (+ 190us) client_negotiation.cc:240] Negotiation successful
0927 12:41:05.120381 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":110,"threads_started":1}
I20260927 12:41:05.122315 4933 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32991
I20260927 12:41:05.122521 4933 heartbeater.cc:464] Registering TS with master...
I20260927 12:41:05.122931 4933 heartbeater.cc:511] Master 127.0.98.62:32991 requested a full tablet report, sending...
I20260927 12:41:05.123833 4477 ts_manager.cc:194] Registered new tserver with Master: 2c0272c693574844aea11c6e416121ed (127.0.98.3:41727)
I20260927 12:41:05.124493 4477 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59649
I20260927 12:41:05.130098 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 12:41:05.139266 4516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:05.132111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42364 (local address 127.0.98.62:32991)
0927 12:41:05.132240 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:05.132244 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:05.132257 (+ 13us) server_negotiation.cc:407] Connection header received
0927 12:41:05.132334 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:05.132337 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:05.132382 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:05.132463 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:05.133154 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:05.133712 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:05.134555 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:05.134777 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:05.134886 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:05.136270 (+ 1384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:05.136274 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:05.136277 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:41:05.136301 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:05.137622 (+ 1321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:05.138038 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:05.138043 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:05.138045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:05.138130 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:05.138393 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:05.138397 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:05.138399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:05.138608 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:05.138747 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:05.138982 (+ 235us) server_negotiation.cc:299] Negotiation successful
0927 12:41:05.139096 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 12:41:05.141994 4477 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42364:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 12:41:05.144997 4477 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:41:05.159112 4944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:05.153244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41985 (local address 127.0.0.1:59468)
0927 12:41:05.154029 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:05.154079 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:05.154209 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:05.154814 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:05.154819 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:05.154848 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:41:05.155021 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:05.155027 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:05.156256 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:05.156261 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:05.157637 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:05.157647 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:05.158572 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:05.158575 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:05.158688 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:05.158723 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 12:41:05.158837 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 12:41:05.158955 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"mutex_wait_us":409,"spinlock_wait_cycles":9344,"thread_start_us":90,"threads_started":1}
I20260927 12:41:05.159116 4945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:05.153390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59468 (local address 127.0.98.1:41985)
0927 12:41:05.153731 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:05.153739 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:05.154352 (+ 613us) server_negotiation.cc:407] Connection header received
0927 12:41:05.154434 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:05.154440 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:05.154657 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:05.154823 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:41:05.155169 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:05.156132 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:05.157757 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:05.158449 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:05.158572 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:05.158720 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:05.158871 (+ 151us) server_negotiation.cc:299] Negotiation successful
0927 12:41:05.158963 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":93,"threads_started":1}
I20260927 12:41:05.159408 4948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:05.154013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39517 (local address 127.0.0.1:45372)
0927 12:41:05.154820 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:05.154832 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:05.154906 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:05.155436 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:05.155439 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:05.155463 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:41:05.155537 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:05.155542 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:05.156851 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:05.156855 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:05.157933 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:05.157944 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:05.159081 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:05.159084 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:05.159157 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:05.159175 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 12:41:05.159238 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 12:41:05.159293 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":755,"mutex_wait_us":30,"thread_start_us":83,"threads_started":1}
I20260927 12:41:05.159745 4947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:05.153244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41727 (local address 127.0.0.1:37410)
0927 12:41:05.154028 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:05.154079 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:05.154210 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:05.155106 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:05.155110 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:05.155130 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:41:05.155210 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:05.155218 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:05.156693 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:05.156697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:05.157851 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:05.157857 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:05.158901 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:05.158904 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:05.158995 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:05.159017 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 12:41:05.159070 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 12:41:05.159135 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"mutex_wait_us":48,"spinlock_wait_cycles":1280,"thread_start_us":59,"threads_started":1}
I20260927 12:41:05.159817 4949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:05.154377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45372 (local address 127.0.98.2:39517)
0927 12:41:05.154685 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:05.154699 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:05.155003 (+ 304us) server_negotiation.cc:407] Connection header received
0927 12:41:05.155101 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:05.155106 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:05.155277 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:05.155444 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:41:05.155656 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:05.156714 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:05.158037 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:05.158761 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:05.158877 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:05.159024 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:05.159573 (+ 549us) server_negotiation.cc:299] Negotiation successful
0927 12:41:05.159675 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":133,"threads_started":1}
I20260927 12:41:05.159909 4946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:05.153421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37410 (local address 127.0.98.3:41727)
0927 12:41:05.154514 (+ 1093us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:05.154527 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:05.154547 (+ 20us) server_negotiation.cc:407] Connection header received
0927 12:41:05.154789 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:05.154794 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:05.154963 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:05.155117 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:41:05.155313 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:05.156475 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:05.157958 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:05.158761 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:05.159497 (+ 736us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:05.159634 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:05.159676 (+ 42us) server_negotiation.cc:299] Negotiation successful
0927 12:41:05.159758 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":192,"threads_started":1}
I20260927 12:41:05.162357 4601 tablet_service.cc:1467] Processing CreateTablet for tablet b89be81ab59049429675da6519df3630 (DEFAULT_TABLE table=test-table [id=3ed0c47a41a646aaa0702c073dc15a1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:41:05.162667 4734 tablet_service.cc:1467] Processing CreateTablet for tablet b89be81ab59049429675da6519df3630 (DEFAULT_TABLE table=test-table [id=3ed0c47a41a646aaa0702c073dc15a1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:41:05.163424 4601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b89be81ab59049429675da6519df3630. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:41:05.163090 4867 tablet_service.cc:1467] Processing CreateTablet for tablet b89be81ab59049429675da6519df3630 (DEFAULT_TABLE table=test-table [id=3ed0c47a41a646aaa0702c073dc15a1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:41:05.163700 4734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b89be81ab59049429675da6519df3630. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:41:05.164064 4867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b89be81ab59049429675da6519df3630. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:41:05.169220 4950 tablet_bootstrap.cc:492] T b89be81ab59049429675da6519df3630 P fd02a2dcafa748e192aff8c7b76fd85b: Bootstrap starting.
I20260927 12:41:05.170997 4951 tablet_bootstrap.cc:492] T b89be81ab59049429675da6519df3630 P 3e11aaa147694132b213c760286d38e2: Bootstrap starting.
I20260927 12:41:05.171336 4952 tablet_bootstrap.cc:492] T b89be81ab59049429675da6519df3630 P 2c0272c693574844aea11c6e416121ed: Bootstrap starting.
I20260927 12:41:05.171495 4950 tablet_bootstrap.cc:654] T b89be81ab59049429675da6519df3630 P fd02a2dcafa748e192aff8c7b76fd85b: Neither blocks nor log segments found. Creating new log.
I20260927 12:41:05.172750 4950 log.cc:824] T b89be81ab59049429675da6519df3630 P fd02a2dcafa748e192aff8c7b76fd85b: Log is configured to *not* fsync() on all Append() calls
I20260927 12:41:05.173038 4951 tablet_bootstrap.cc:654] T b89be81ab59049429675da6519df3630 P 3e11aaa147694132b213c760286d38e2: Neither blocks nor log segments found. Creating new log.
I20260927 12:41:05.173389 4952 tablet_bootstrap.cc:654] T b89be81ab59049429675da6519df3630 P 2c0272c693574844aea11c6e416121ed: Neither blocks nor log segments found. Creating new log.
I20260927 12:41:05.174129 4951 log.cc:824] T b89be81ab59049429675da6519df3630 P 3e11aaa147694132b213c760286d38e2: Log is configured to *not* fsync() on all Append() calls
I20260927 12:41:05.174614 4952 log.cc:824] T b89be81ab59049429675da6519df3630 P 2c0272c693574844aea11c6e416121ed: Log is configured to *not* fsync() on all Append() calls
I20260927 12:41:05.174846 4950 tablet_bootstrap.cc:492] T b89be81ab59049429675da6519df3630 P fd02a2dcafa748e192aff8c7b76fd85b: No bootstrap required, opened a new log
I20260927 12:41:05.175029 4950 ts_tablet_manager.cc:1397] T b89be81ab59049429675da6519df3630 P fd02a2dcafa748e192aff8c7b76fd85b: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260927 12:41:05.175837 4951 tablet_bootstrap.cc:492] T b89be81ab59049429675da6519df3630 P 3e11aaa147694132b213c760286d38e2: No bootstrap required, opened a new log
I20260927 12:41:05.175977 4951 ts_tablet_manager.cc:1397] T b89be81ab59049429675da6519df3630 P 3e11aaa147694132b213c760286d38e2: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 12:41:05.177865 4950 raft_consensus.cc:348] T b89be81ab59049429675da6519df3630 P fd02a2dcafa748e192aff8c7b76fd85b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd02a2dcafa748e192aff8c7b76fd85b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41985 } } peers { permanent_uuid: "2c0272c693574844aea11c6e416121ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41727 } } peers { permanent_uuid: "3e11aaa147694132b213c760286d38e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39517 } }
I20260927 12:41:05.178095 4950 raft_consensus.cc:374] T b89be81ab59049429675da6519df3630 P fd02a2dcafa748e192aff8c7b76fd85b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:41:05.178174 4950 raft_consensus.cc:729] T b89be81ab59049429675da6519df3630 P fd02a2dcafa748e192aff8c7b76fd85b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd02a2dcafa748e192aff8c7b76fd85b, State: Initialized, Role: FOLLOWER
I20260927 12:41:05.178637 4950 consensus_queue.cc:260] T b89be81ab59049429675da6519df3630 P fd02a2dcafa748e192aff8c7b76fd85b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd02a2dcafa748e192aff8c7b76fd85b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41985 } } peers { permanent_uuid: "2c0272c693574844aea11c6e416121ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41727 } } peers { permanent_uuid: "3e11aaa147694132b213c760286d38e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39517 } }
I20260927 12:41:05.179246 4667 heartbeater.cc:503] Master 127.0.98.62:32991 was elected leader, sending a full tablet report...
I20260927 12:41:05.179270 4952 tablet_bootstrap.cc:492] T b89be81ab59049429675da6519df3630 P 2c0272c693574844aea11c6e416121ed: No bootstrap required, opened a new log
I20260927 12:41:05.179185 4951 raft_consensus.cc:348] T b89be81ab59049429675da6519df3630 P 3e11aaa147694132b213c760286d38e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd02a2dcafa748e192aff8c7b76fd85b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41985 } } peers { permanent_uuid: "2c0272c693574844aea11c6e416121ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41727 } } peers { permanent_uuid: "3e11aaa147694132b213c760286d38e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39517 } }
I20260927 12:41:05.179440 4952 ts_tablet_manager.cc:1397] T b89be81ab59049429675da6519df3630 P 2c0272c693574844aea11c6e416121ed: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.004s
I20260927 12:41:05.179450 4951 raft_consensus.cc:374] T b89be81ab59049429675da6519df3630 P 3e11aaa147694132b213c760286d38e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:41:05.179508 4951 raft_consensus.cc:729] T b89be81ab59049429675da6519df3630 P 3e11aaa147694132b213c760286d38e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e11aaa147694132b213c760286d38e2, State: Initialized, Role: FOLLOWER
I20260927 12:41:05.179960 4951 consensus_queue.cc:260] T b89be81ab59049429675da6519df3630 P 3e11aaa147694132b213c760286d38e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd02a2dcafa748e192aff8c7b76fd85b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41985 } } peers { permanent_uuid: "2c0272c693574844aea11c6e416121ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41727 } } peers { permanent_uuid: "3e11aaa147694132b213c760286d38e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39517 } }
I20260927 12:41:05.180996 4951 ts_tablet_manager.cc:1428] T b89be81ab59049429675da6519df3630 P 3e11aaa147694132b213c760286d38e2: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 12:41:05.181371 4800 heartbeater.cc:503] Master 127.0.98.62:32991 was elected leader, sending a full tablet report...
I20260927 12:41:05.182180 4950 ts_tablet_manager.cc:1428] T b89be81ab59049429675da6519df3630 P fd02a2dcafa748e192aff8c7b76fd85b: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20260927 12:41:05.183285 4952 raft_consensus.cc:348] T b89be81ab59049429675da6519df3630 P 2c0272c693574844aea11c6e416121ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd02a2dcafa748e192aff8c7b76fd85b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41985 } } peers { permanent_uuid: "2c0272c693574844aea11c6e416121ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41727 } } peers { permanent_uuid: "3e11aaa147694132b213c760286d38e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39517 } }
I20260927 12:41:05.183581 4952 raft_consensus.cc:374] T b89be81ab59049429675da6519df3630 P 2c0272c693574844aea11c6e416121ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:41:05.183672 4952 raft_consensus.cc:729] T b89be81ab59049429675da6519df3630 P 2c0272c693574844aea11c6e416121ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c0272c693574844aea11c6e416121ed, State: Initialized, Role: FOLLOWER
I20260927 12:41:05.184130 4952 consensus_queue.cc:260] T b89be81ab59049429675da6519df3630 P 2c0272c693574844aea11c6e416121ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd02a2dcafa748e192aff8c7b76fd85b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41985 } } peers { permanent_uuid: "2c0272c693574844aea11c6e416121ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41727 } } peers { permanent_uuid: "3e11aaa147694132b213c760286d38e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39517 } }
I20260927 12:41:05.184847 4933 heartbeater.cc:503] Master 127.0.98.62:32991 was elected leader, sending a full tablet report...
I20260927 12:41:05.184938 4952 ts_tablet_manager.cc:1428] T b89be81ab59049429675da6519df3630 P 2c0272c693574844aea11c6e416121ed: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 12:41:05.294581 4956 raft_consensus.cc:482] T b89be81ab59049429675da6519df3630 P fd02a2dcafa748e192aff8c7b76fd85b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:41:05.294829 4956 raft_consensus.cc:504] T b89be81ab59049429675da6519df3630 P fd02a2dcafa748e192aff8c7b76fd85b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd02a2dcafa748e192aff8c7b76fd85b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41985 } } peers { permanent_uuid: "2c0272c693574844aea11c6e416121ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41727 } } peers { permanent_uuid: "3e11aaa147694132b213c760286d38e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39517 } }
I20260927 12:41:05.296011 4956 leader_election.cc:288] T b89be81ab59049429675da6519df3630 P fd02a2dcafa748e192aff8c7b76fd85b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2c0272c693574844aea11c6e416121ed (127.0.98.3:41727), 3e11aaa147694132b213c760286d38e2 (127.0.98.2:39517)
I20260927 12:41:05.299983 4949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:05.296439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43419 (local address 127.0.98.2:39517)
0927 12:41:05.296578 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:05.296583 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:05.296772 (+ 189us) server_negotiation.cc:407] Connection header received
0927 12:41:05.296922 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:05.296927 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:05.296975 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:05.297059 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:41:05.297313 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:05.297853 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:05.298972 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:05.299363 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:05.299478 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:05.299541 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:05.299781 (+ 240us) server_negotiation.cc:299] Negotiation successful
0927 12:41:05.299845 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 12:41:05.299988 4946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:05.296366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45429 (local address 127.0.98.3:41727)
0927 12:41:05.296496 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:05.296502 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:05.296676 (+ 174us) server_negotiation.cc:407] Connection header received
0927 12:41:05.296852 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:05.296860 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:05.296903 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:05.296988 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:41:05.297274 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:05.297793 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:05.298982 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:05.299379 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:05.299497 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:05.299560 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:05.299797 (+ 237us) server_negotiation.cc:299] Negotiation successful
0927 12:41:05.299862 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 12:41:05.299991 4959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:05.296274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39517 (local address 127.0.98.1:43419)
0927 12:41:05.296661 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:05.296673 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:05.296810 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:05.297066 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:05.297069 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:05.297083 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:41:05.297205 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:05.297211 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:05.297954 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:05.297961 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:05.298855 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:05.298863 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:05.299479 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:05.299483 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:05.299648 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:05.299673 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 12:41:05.299765 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 12:41:05.299831 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":73,"thread_start_us":110,"threads_started":1}
I20260927 12:41:05.300004 4960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:05.296216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41727 (local address 127.0.98.1:45429)
0927 12:41:05.296577 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:05.296589 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:05.296732 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:05.297007 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:05.297010 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:05.297029 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:41:05.297155 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:05.297160 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:05.297902 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:05.297908 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:05.298859 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:05.298868 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:05.299499 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:05.299502 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:05.299669 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:05.299686 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 12:41:05.299780 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 12:41:05.299847 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"mutex_wait_us":87,"thread_start_us":102,"threads_started":1}
I20260927 12:41:05.300663 4754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b89be81ab59049429675da6519df3630" candidate_uuid: "fd02a2dcafa748e192aff8c7b76fd85b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e11aaa147694132b213c760286d38e2" is_pre_election: true
I20260927 12:41:05.300662 4887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b89be81ab59049429675da6519df3630" candidate_uuid: "fd02a2dcafa748e192aff8c7b76fd85b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c0272c693574844aea11c6e416121ed" is_pre_election: true
I20260927 12:41:05.300963 4754 raft_consensus.cc:2454] T b89be81ab59049429675da6519df3630 P 3e11aaa147694132b213c760286d38e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd02a2dcafa748e192aff8c7b76fd85b in term 0.
I20260927 12:41:05.300963 4887 raft_consensus.cc:2454] T b89be81ab59049429675da6519df3630 P 2c0272c693574844aea11c6e416121ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd02a2dcafa748e192aff8c7b76fd85b in term 0.
I20260927 12:41:05.301494 4554 leader_election.cc:302] T b89be81ab59049429675da6519df3630 P fd02a2dcafa748e192aff8c7b76fd85b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c0272c693574844aea11c6e416121ed, fd02a2dcafa748e192aff8c7b76fd85b; no voters:
I20260927 12:41:05.301831 4956 raft_consensus.cc:2781] T b89be81ab59049429675da6519df3630 P fd02a2dcafa748e192aff8c7b76fd85b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:41:05.301915 4956 raft_consensus.cc:482] T b89be81ab59049429675da6519df3630 P fd02a2dcafa748e192aff8c7b76fd85b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:41:05.301957 4956 raft_consensus.cc:3037] T b89be81ab59049429675da6519df3630 P fd02a2dcafa748e192aff8c7b76fd85b [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:41:05.302946 4956 raft_consensus.cc:504] T b89be81ab59049429675da6519df3630 P fd02a2dcafa748e192aff8c7b76fd85b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd02a2dcafa748e192aff8c7b76fd85b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41985 } } peers { permanent_uuid: "2c0272c693574844aea11c6e416121ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41727 } } peers { permanent_uuid: "3e11aaa147694132b213c760286d38e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39517 } }
I20260927 12:41:05.303282 4956 leader_election.cc:288] T b89be81ab59049429675da6519df3630 P fd02a2dcafa748e192aff8c7b76fd85b [CANDIDATE]: Term 1 election: Requested vote from peers 2c0272c693574844aea11c6e416121ed (127.0.98.3:41727), 3e11aaa147694132b213c760286d38e2 (127.0.98.2:39517)
I20260927 12:41:05.303551 4887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b89be81ab59049429675da6519df3630" candidate_uuid: "fd02a2dcafa748e192aff8c7b76fd85b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c0272c693574844aea11c6e416121ed"
I20260927 12:41:05.303658 4754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b89be81ab59049429675da6519df3630" candidate_uuid: "fd02a2dcafa748e192aff8c7b76fd85b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e11aaa147694132b213c760286d38e2"
I20260927 12:41:05.303695 4887 raft_consensus.cc:3037] T b89be81ab59049429675da6519df3630 P 2c0272c693574844aea11c6e416121ed [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:41:05.303771 4754 raft_consensus.cc:3037] T b89be81ab59049429675da6519df3630 P 3e11aaa147694132b213c760286d38e2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:41:05.305058 4887 raft_consensus.cc:2454] T b89be81ab59049429675da6519df3630 P 2c0272c693574844aea11c6e416121ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd02a2dcafa748e192aff8c7b76fd85b in term 1.
I20260927 12:41:05.305058 4754 raft_consensus.cc:2454] T b89be81ab59049429675da6519df3630 P 3e11aaa147694132b213c760286d38e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd02a2dcafa748e192aff8c7b76fd85b in term 1.
I20260927 12:41:05.305352 4557 leader_election.cc:302] T b89be81ab59049429675da6519df3630 P fd02a2dcafa748e192aff8c7b76fd85b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2c0272c693574844aea11c6e416121ed, 3e11aaa147694132b213c760286d38e2, fd02a2dcafa748e192aff8c7b76fd85b; no voters:
I20260927 12:41:05.305511 4956 raft_consensus.cc:2781] T b89be81ab59049429675da6519df3630 P fd02a2dcafa748e192aff8c7b76fd85b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:41:05.305790 4956 raft_consensus.cc:686] T b89be81ab59049429675da6519df3630 P fd02a2dcafa748e192aff8c7b76fd85b [term 1 LEADER]: Becoming Leader. State: Replica: fd02a2dcafa748e192aff8c7b76fd85b, State: Running, Role: LEADER
I20260927 12:41:05.306113 4956 consensus_queue.cc:237] T b89be81ab59049429675da6519df3630 P fd02a2dcafa748e192aff8c7b76fd85b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd02a2dcafa748e192aff8c7b76fd85b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41985 } } peers { permanent_uuid: "2c0272c693574844aea11c6e416121ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41727 } } peers { permanent_uuid: "3e11aaa147694132b213c760286d38e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39517 } }
I20260927 12:41:05.309079 4477 catalog_manager.cc:5495] T b89be81ab59049429675da6519df3630 P fd02a2dcafa748e192aff8c7b76fd85b reported cstate change: term changed from 0 to 1, leader changed from <none> to fd02a2dcafa748e192aff8c7b76fd85b (127.0.98.1). New cstate: current_term: 1 leader_uuid: "fd02a2dcafa748e192aff8c7b76fd85b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd02a2dcafa748e192aff8c7b76fd85b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41985 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c0272c693574844aea11c6e416121ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e11aaa147694132b213c760286d38e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39517 } health_report { overall_health: UNKNOWN } } }
I20260927 12:41:05.351030 4945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:05.347387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59484 (local address 127.0.98.1:41985)
0927 12:41:05.347511 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:05.347515 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:05.347528 (+ 13us) server_negotiation.cc:407] Connection header received
0927 12:41:05.347649 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:05.347652 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:05.347707 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:05.347799 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:41:05.348009 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:05.348586 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:05.349493 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:05.349786 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:05.349898 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:05.350201 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:41:05.350622 (+ 421us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:41:05.350758 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:05.350823 (+ 65us) server_negotiation.cc:299] Negotiation successful
0927 12:41:05.350876 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 12:41:05.356630 4887 raft_consensus.cc:1261] T b89be81ab59049429675da6519df3630 P 2c0272c693574844aea11c6e416121ed [term 1 FOLLOWER]: Refusing update from remote peer fd02a2dcafa748e192aff8c7b76fd85b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:41:05.356637 4754 raft_consensus.cc:1261] T b89be81ab59049429675da6519df3630 P 3e11aaa147694132b213c760286d38e2 [term 1 FOLLOWER]: Refusing update from remote peer fd02a2dcafa748e192aff8c7b76fd85b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:41:05.357307 4956 consensus_queue.cc:1035] T b89be81ab59049429675da6519df3630 P fd02a2dcafa748e192aff8c7b76fd85b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c0272c693574844aea11c6e416121ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:41:05.357524 4961 consensus_queue.cc:1035] T b89be81ab59049429675da6519df3630 P fd02a2dcafa748e192aff8c7b76fd85b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e11aaa147694132b213c760286d38e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 12:41:05.357683 4934 tablet.cc:2369] T b89be81ab59049429675da6519df3630 P 2c0272c693574844aea11c6e416121ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:41:05.363775 4965 mvcc.cc:205] Tried to move back new op lower bound from 7333940696493826048 to 7333940696296370176. Current Snapshot: MvccSnapshot[applied={T|T < 7333940696493826048}]
I20260927 12:41:05.367612 4964 mvcc.cc:205] Tried to move back new op lower bound from 7333940696493826048 to 7333940696296370176. Current Snapshot: MvccSnapshot[applied={T|T < 7333940696493826048 or (T in {7333940696493826048})}]
I20260927 12:41:05.368055 4968 mvcc.cc:205] Tried to move back new op lower bound from 7333940696493826048 to 7333940696296370176. Current Snapshot: MvccSnapshot[applied={T|T < 7333940696493826048}]
I20260927 12:41:05.371909 4477 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42364:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:41:05.372036 4477 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42364:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:41:05.374270 4477 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8c56efcb986546e09a7c3265f21e86a1: Sending DeleteTablet for 3 replicas of tablet b89be81ab59049429675da6519df3630
I20260927 12:41:05.374958 4734 tablet_service.cc:1514] Processing DeleteTablet for tablet b89be81ab59049429675da6519df3630 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:41:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45372
I20260927 12:41:05.374984 4867 tablet_service.cc:1514] Processing DeleteTablet for tablet b89be81ab59049429675da6519df3630 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:41:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37410
I20260927 12:41:05.375416 4601 tablet_service.cc:1514] Processing DeleteTablet for tablet b89be81ab59049429675da6519df3630 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:41:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59468
I20260927 12:41:05.375847 4974 tablet_replica.cc:330] T b89be81ab59049429675da6519df3630 P 3e11aaa147694132b213c760286d38e2: stopping tablet replica
I20260927 12:41:05.376111 4974 raft_consensus.cc:2229] T b89be81ab59049429675da6519df3630 P 3e11aaa147694132b213c760286d38e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:41:05.376384 4974 raft_consensus.cc:2258] T b89be81ab59049429675da6519df3630 P 3e11aaa147694132b213c760286d38e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:41:05.376677 4975 tablet_replica.cc:330] T b89be81ab59049429675da6519df3630 P fd02a2dcafa748e192aff8c7b76fd85b: stopping tablet replica
I20260927 12:41:05.376816 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 4539
I20260927 12:41:05.379770 4974 ts_tablet_manager.cc:1905] T b89be81ab59049429675da6519df3630 P 3e11aaa147694132b213c760286d38e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:41:05.382372 4974 ts_tablet_manager.cc:1918] T b89be81ab59049429675da6519df3630 P 3e11aaa147694132b213c760286d38e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:41:05.382476 4974 log.cc:1196] T b89be81ab59049429675da6519df3630 P 3e11aaa147694132b213c760286d38e2: Deleting WAL directory at /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/minicluster-data/ts-1/wal/wals/b89be81ab59049429675da6519df3630
I20260927 12:41:05.382848 4974 ts_tablet_manager.cc:1939] T b89be81ab59049429675da6519df3630 P 3e11aaa147694132b213c760286d38e2: Deleting consensus metadata
I20260927 12:41:05.384132 4464 catalog_manager.cc:4841] TS 3e11aaa147694132b213c760286d38e2 (127.0.98.2:39517): tablet b89be81ab59049429675da6519df3630 (table test-table [id=3ed0c47a41a646aaa0702c073dc15a1e]) successfully deleted
I20260927 12:41:05.384171 4973 tablet_replica.cc:330] T b89be81ab59049429675da6519df3630 P 2c0272c693574844aea11c6e416121ed: stopping tablet replica
I20260927 12:41:05.384486 4973 raft_consensus.cc:2229] T b89be81ab59049429675da6519df3630 P 2c0272c693574844aea11c6e416121ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:41:05.384801 4973 raft_consensus.cc:2258] T b89be81ab59049429675da6519df3630 P 2c0272c693574844aea11c6e416121ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:41:05.386165 4973 ts_tablet_manager.cc:1905] T b89be81ab59049429675da6519df3630 P 2c0272c693574844aea11c6e416121ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:41:05.387679 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 4672
W20260927 12:41:05.388757 4464 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41985 (error 108)
I20260927 12:41:05.389379 4973 ts_tablet_manager.cc:1918] T b89be81ab59049429675da6519df3630 P 2c0272c693574844aea11c6e416121ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:41:05.389465 4973 log.cc:1196] T b89be81ab59049429675da6519df3630 P 2c0272c693574844aea11c6e416121ed: Deleting WAL directory at /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790512840836255-392-0/minicluster-data/ts-2/wal/wals/b89be81ab59049429675da6519df3630
I20260927 12:41:05.389822 4973 ts_tablet_manager.cc:1939] T b89be81ab59049429675da6519df3630 P 2c0272c693574844aea11c6e416121ed: Deleting consensus metadata
I20260927 12:41:05.389892 4947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:05.389586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41985 (local address 127.0.0.1:59500)
0927 12:41:05.389748 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:05.389821 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260927 12:41:05.390023 4464 catalog_manager.cc:4584] TS fd02a2dcafa748e192aff8c7b76fd85b (127.0.98.1:41985): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b89be81ab59049429675da6519df3630: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41985: connect: Connection refused (error 111)
I20260927 12:41:05.391000 4461 catalog_manager.cc:4841] TS 2c0272c693574844aea11c6e416121ed (127.0.98.3:41727): tablet b89be81ab59049429675da6519df3630 (table test-table [id=3ed0c47a41a646aaa0702c073dc15a1e]) successfully deleted
I20260927 12:41:05.394541 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 4805
I20260927 12:41:05.400691 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 4446
2026-09-27T12:41:05Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2373 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-27T12:41:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:41:05Z Disabled control of system clock
I20260927 12:41:05.430130 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512840836255-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512840836255-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512840836255-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46117
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46117
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 12:41:05.537916 4980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:41:05.538194 4980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:41:05.538232 4980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:41:05.542487 4980 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:41:05.542557 4980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:41:05.542572 4980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:41:05.542590 4980 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:41:05.542605 4980 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:41:05.547770 4980 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512840836255-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512840836255-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46117
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4980
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512840836255-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:41:05.548771 4980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:41:05.550127 4980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:41:05.558072 4986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:05.558080 4985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:05.558168 4988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:05.558204 4980 server_base.cc:1034] running on GCE node
I20260927 12:41:05.558894 4980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:41:05.559967 4980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:41:05.561127 4980 hybrid_clock.cc:648] HybridClock initialized: now 1790512865561109 us; error 29 us; skew 500 ppm
I20260927 12:41:05.563127 4980 webserver.cc:460] Webserver started at http://127.0.98.62:38517/ using document root <none> and password file <none>
I20260927 12:41:05.563668 4980 fs_manager.cc:359] Metadata directory not provided
I20260927 12:41:05.563728 4980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:41:05.563947 4980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:41:05.566004 4980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512840836255-392-0/minicluster-data/master-0/data/instance:
uuid: "04bcfe09cab24583aad7cd6c4e551461"
format_stamp: "Formatted at 2026-09-27 12:41:05 on dist-test-slave-bnc1"
server_key: "61c16adfc03376e1dc08076f77254ab5"
server_key_iv: "6b45703972077a1b30a17ea8a72defd7"
server_key_version: "encryptionkey@0"
I20260927 12:41:05.566545 4980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512840836255-392-0/minicluster-data/master-0/wal/instance:
uuid: "04bcfe09cab24583aad7cd6c4e551461"
format_stamp: "Formatted at 2026-09-27 12:41:05 on dist-test-slave-bnc1"
server_key: "61c16adfc03376e1dc08076f77254ab5"
server_key_iv: "6b45703972077a1b30a17ea8a72defd7"
server_key_version: "encryptionkey@0"
I20260927 12:41:05.569970 4980 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 12:41:05.572283 4994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:05.573423 4980 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:41:05.573553 4980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512840836255-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512840836255-392-0/minicluster-data/master-0/wal
uuid: "04bcfe09cab24583aad7cd6c4e551461"
format_stamp: "Formatted at 2026-09-27 12:41:05 on dist-test-slave-bnc1"
server_key: "61c16adfc03376e1dc08076f77254ab5"
server_key_iv: "6b45703972077a1b30a17ea8a72defd7"
server_key_version: "encryptionkey@0"
I20260927 12:41:05.573712 4980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512840836255-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512840836255-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512840836255-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:41:05.578135 4980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:41:05.578718 4980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:41:05.578890 4980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:41:05.585451 4980 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46117
I20260927 12:41:05.585457 5046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46117 every 8 connection(s)
I20260927 12:41:05.586404 4980 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:41:05.589457 5047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:41:05.595079 5047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04bcfe09cab24583aad7cd6c4e551461: Bootstrap starting.
I20260927 12:41:05.595345 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 4980
I20260927 12:41:05.595479 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512840836255-392-0/minicluster-data/master-0/wal/instance
I20260927 12:41:05.595732 392 external_mini_cluster.cc:1428] Setting key 4beb40f5ea195ccbf6222d455d0f609f
I20260927 12:41:05.597764 5047 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04bcfe09cab24583aad7cd6c4e551461: Neither blocks nor log segments found. Creating new log.
I20260927 12:41:05.598518 5047 log.cc:824] T 00000000000000000000000000000000 P 04bcfe09cab24583aad7cd6c4e551461: Log is configured to *not* fsync() on all Append() calls
I20260927 12:41:05.600574 5047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04bcfe09cab24583aad7cd6c4e551461: No bootstrap required, opened a new log
I20260927 12:41:05.601298 5050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:05.597032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59820 (local address 127.0.98.62:46117)
0927 12:41:05.597729 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:05.597744 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:05.597779 (+ 35us) server_negotiation.cc:407] Connection header received
0927 12:41:05.598481 (+ 702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:05.598498 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:05.598895 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:05.599226 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:05.600019 (+ 793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:05.600023 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:05.600035 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:41:05.600045 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:05.600098 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:41:05.600113 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:05.600227 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:05.600343 (+ 116us) server_negotiation.cc:299] Negotiation successful
0927 12:41:05.600580 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":139,"threads_started":1}
I20260927 12:41:05.604641 5047 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04bcfe09cab24583aad7cd6c4e551461 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04bcfe09cab24583aad7cd6c4e551461" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46117 } }
I20260927 12:41:05.604920 5047 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04bcfe09cab24583aad7cd6c4e551461 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:41:05.604992 5047 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04bcfe09cab24583aad7cd6c4e551461 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04bcfe09cab24583aad7cd6c4e551461, State: Initialized, Role: FOLLOWER
I20260927 12:41:05.605528 5047 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04bcfe09cab24583aad7cd6c4e551461 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04bcfe09cab24583aad7cd6c4e551461" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46117 } }
I20260927 12:41:05.605708 5047 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04bcfe09cab24583aad7cd6c4e551461 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:41:05.605767 5047 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04bcfe09cab24583aad7cd6c4e551461 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:41:05.605839 5047 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04bcfe09cab24583aad7cd6c4e551461 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:41:05.606674 5047 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04bcfe09cab24583aad7cd6c4e551461 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04bcfe09cab24583aad7cd6c4e551461" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46117 } }
I20260927 12:41:05.606968 5047 leader_election.cc:302] T 00000000000000000000000000000000 P 04bcfe09cab24583aad7cd6c4e551461 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04bcfe09cab24583aad7cd6c4e551461; no voters:
I20260927 12:41:05.607219 5047 leader_election.cc:288] T 00000000000000000000000000000000 P 04bcfe09cab24583aad7cd6c4e551461 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:41:05.607328 5052 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04bcfe09cab24583aad7cd6c4e551461 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:41:05.607559 5052 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04bcfe09cab24583aad7cd6c4e551461 [term 1 LEADER]: Becoming Leader. State: Replica: 04bcfe09cab24583aad7cd6c4e551461, State: Running, Role: LEADER
I20260927 12:41:05.607903 5052 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04bcfe09cab24583aad7cd6c4e551461 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04bcfe09cab24583aad7cd6c4e551461" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46117 } }
I20260927 12:41:05.608489 5047 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04bcfe09cab24583aad7cd6c4e551461 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:41:05.609716 5053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04bcfe09cab24583aad7cd6c4e551461 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04bcfe09cab24583aad7cd6c4e551461" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04bcfe09cab24583aad7cd6c4e551461" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46117 } } }
I20260927 12:41:05.609704 5054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04bcfe09cab24583aad7cd6c4e551461 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04bcfe09cab24583aad7cd6c4e551461. Latest consensus state: current_term: 1 leader_uuid: "04bcfe09cab24583aad7cd6c4e551461" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04bcfe09cab24583aad7cd6c4e551461" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46117 } } }
I20260927 12:41:05.609840 5053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04bcfe09cab24583aad7cd6c4e551461 [sys.catalog]: This master's current role is: LEADER
I20260927 12:41:05.609840 5054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04bcfe09cab24583aad7cd6c4e551461 [sys.catalog]: This master's current role is: LEADER
I20260927 12:41:05.610206 5061 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:41:05.612877 5061 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:41:05.617708 5061 catalog_manager.cc:1347] Generated new cluster ID: a0b6910045974ee28a46a0db0c818c3b
I20260927 12:41:05.617775 5061 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:41:05.635273 5061 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:41:05.635658 5061 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:41:05.642861 5061 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04bcfe09cab24583aad7cd6c4e551461: Generated new TSK 0
I20260927 12:41:05.643720 5061 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:41:05.648738 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512840836255-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512840836255-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512840836255-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46117
--builtin_ntp_servers=127.0.98.20:38093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 12:41:05.759529 5071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:41:05.759796 5071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:41:05.759841 5071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:41:05.763904 5071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:41:05.763964 5071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:41:05.764025 5071 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:41:05.768770 5071 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512840836255-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512840836255-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5071
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512840836255-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:41:05.769874 5071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:41:05.771245 5071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:41:05.780884 5077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:05.780910 5079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:05.781026 5071 server_base.cc:1034] running on GCE node
W20260927 12:41:05.781194 5076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:05.781643 5071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:41:05.782289 5071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:41:05.783459 5071 hybrid_clock.cc:648] HybridClock initialized: now 1790512865783427 us; error 45 us; skew 500 ppm
I20260927 12:41:05.785219 5071 webserver.cc:460] Webserver started at http://127.0.98.1:39361/ using document root <none> and password file <none>
I20260927 12:41:05.785820 5071 fs_manager.cc:359] Metadata directory not provided
I20260927 12:41:05.785881 5071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:41:05.786126 5071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:41:05.788168 5071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512840836255-392-0/minicluster-data/ts-0/data/instance:
uuid: "a4d6279803074fb290c0c0e298a275ea"
format_stamp: "Formatted at 2026-09-27 12:41:05 on dist-test-slave-bnc1"
server_key: "ddd8cbd7bf0b1b04f61e19cd22ae86e9"
server_key_iv: "6642883741e598b53391446f759792d5"
server_key_version: "encryptionkey@0"
I20260927 12:41:05.788703 5071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512840836255-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a4d6279803074fb290c0c0e298a275ea"
format_stamp: "Formatted at 2026-09-27 12:41:05 on dist-test-slave-bnc1"
server_key: "ddd8cbd7bf0b1b04f61e19cd22ae86e9"
server_key_iv: "6642883741e598b53391446f759792d5"
server_key_version: "encryptionkey@0"
I20260927 12:41:05.792280 5071 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 12:41:05.794505 5085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:05.795655 5071 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 12:41:05.795780 5071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512840836255-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512840836255-392-0/minicluster-data/ts-0/wal
uuid: "a4d6279803074fb290c0c0e298a275ea"
format_stamp: "Formatted at 2026-09-27 12:41:05 on dist-test-slave-bnc1"
server_key: "ddd8cbd7bf0b1b04f61e19cd22ae86e9"
server_key_iv: "6642883741e598b53391446f759792d5"
server_key_version: "encryptionkey@0"
I20260927 12:41:05.795881 5071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512840836255-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512840836255-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512840836255-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:41:05.800158 5071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:41:05.800694 5071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:41:05.800861 5071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:41:05.801362 5071 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:41:05.802229 5071 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:41:05.802297 5071 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:05.802362 5071 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:41:05.802412 5071 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:05.810925 5071 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41277
I20260927 12:41:05.810950 5198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41277 every 8 connection(s)
I20260927 12:41:05.811736 5071 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:41:05.814827 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 5071
I20260927 12:41:05.814955 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512840836255-392-0/minicluster-data/ts-0/wal/instance
I20260927 12:41:05.815244 392 external_mini_cluster.cc:1428] Setting key f7f2e1fd9521312edc3433e70884acc3
I20260927 12:41:05.817205 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512840836255-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512840836255-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512840836255-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46117
--builtin_ntp_servers=127.0.98.20:38093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 12:41:05.818034 5050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:05.813435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37831 (local address 127.0.98.62:46117)
0927 12:41:05.813609 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:05.813614 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:05.814634 (+ 1020us) server_negotiation.cc:407] Connection header received
0927 12:41:05.816102 (+ 1468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:05.816106 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:05.816140 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:05.816229 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:05.817539 (+ 1310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:05.817543 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:05.817546 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:41:05.817558 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:05.817601 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:41:05.817612 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:05.817701 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:05.817862 (+ 161us) server_negotiation.cc:299] Negotiation successful
0927 12:41:05.817907 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 12:41:05.819187 5201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:05.813848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46117 (local address 127.0.98.1:37831)
0927 12:41:05.814446 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:05.814483 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:05.815883 (+ 1400us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:05.816457 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:05.816468 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:05.816585 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:05.817211 (+ 626us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 12:41:05.817213 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:05.817275 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 12:41:05.817381 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:41:05.817698 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:41:05.817708 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:41:05.817715 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 12:41:05.817903 (+ 188us) client_negotiation.cc:240] Negotiation successful
0927 12:41:05.818142 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":101,"threads_started":1}
I20260927 12:41:05.820425 5199 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46117
I20260927 12:41:05.820657 5199 heartbeater.cc:464] Registering TS with master...
I20260927 12:41:05.821115 5199 heartbeater.cc:511] Master 127.0.98.62:46117 requested a full tablet report, sending...
I20260927 12:41:05.822770 5011 ts_manager.cc:194] Registered new tserver with Master: a4d6279803074fb290c0c0e298a275ea (127.0.98.1:41277)
W20260927 12:41:05.930178 5202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:41:05.930447 5202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:41:05.930487 5202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:41:05.934592 5202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:41:05.934648 5202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:41:05.934729 5202 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:41:05.939446 5202 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512840836255-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512840836255-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5202
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512840836255-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:41:05.940517 5202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:41:05.941979 5202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:41:05.951413 5208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:05.951426 5207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:05.951467 5202 server_base.cc:1034] running on GCE node
W20260927 12:41:05.951591 5210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:05.952044 5202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:41:05.952724 5202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:41:05.953902 5202 hybrid_clock.cc:648] HybridClock initialized: now 1790512865953880 us; error 58 us; skew 500 ppm
I20260927 12:41:05.955724 5202 webserver.cc:460] Webserver started at http://127.0.98.2:39113/ using document root <none> and password file <none>
I20260927 12:41:05.956288 5202 fs_manager.cc:359] Metadata directory not provided
I20260927 12:41:05.956374 5202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:41:05.956607 5202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:41:05.958696 5202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512840836255-392-0/minicluster-data/ts-1/data/instance:
uuid: "8ce7cc6d39fa49e2b5ad2da45f45977a"
format_stamp: "Formatted at 2026-09-27 12:41:05 on dist-test-slave-bnc1"
server_key: "38f013573b60be73665eaa8261023cd0"
server_key_iv: "0fdf859230e1e88b6d5ab18cf1cbfec7"
server_key_version: "encryptionkey@0"
I20260927 12:41:05.959260 5202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512840836255-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8ce7cc6d39fa49e2b5ad2da45f45977a"
format_stamp: "Formatted at 2026-09-27 12:41:05 on dist-test-slave-bnc1"
server_key: "38f013573b60be73665eaa8261023cd0"
server_key_iv: "0fdf859230e1e88b6d5ab18cf1cbfec7"
server_key_version: "encryptionkey@0"
I20260927 12:41:05.962875 5202 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 12:41:05.965288 5216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:05.966611 5202 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:41:05.966747 5202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512840836255-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512840836255-392-0/minicluster-data/ts-1/wal
uuid: "8ce7cc6d39fa49e2b5ad2da45f45977a"
format_stamp: "Formatted at 2026-09-27 12:41:05 on dist-test-slave-bnc1"
server_key: "38f013573b60be73665eaa8261023cd0"
server_key_iv: "0fdf859230e1e88b6d5ab18cf1cbfec7"
server_key_version: "encryptionkey@0"
I20260927 12:41:05.966858 5202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512840836255-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512840836255-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512840836255-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:41:05.971374 5202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:41:05.971946 5202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:41:05.972123 5202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:41:05.972651 5202 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:41:05.973510 5202 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:41:05.973578 5202 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:05.973685 5202 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:41:05.973734 5202 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:05.982431 5202 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42189
I20260927 12:41:05.982503 5329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42189 every 8 connection(s)
I20260927 12:41:05.983330 5202 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:41:05.983671 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 5202
I20260927 12:41:05.983801 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512840836255-392-0/minicluster-data/ts-1/wal/instance
I20260927 12:41:05.984150 392 external_mini_cluster.cc:1428] Setting key 12da397d114a94594c7480a84b2816fa
I20260927 12:41:05.986191 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512840836255-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512840836255-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512840836255-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46117
--builtin_ntp_servers=127.0.98.20:38093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 12:41:05.990152 5050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:05.985358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52645 (local address 127.0.98.62:46117)
0927 12:41:05.985485 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:05.985489 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:05.986709 (+ 1220us) server_negotiation.cc:407] Connection header received
0927 12:41:05.988259 (+ 1550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:05.988262 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:05.988297 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:05.988368 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:05.989614 (+ 1246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:05.989621 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:05.989626 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:41:05.989644 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:05.989670 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:41:05.989683 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:05.989779 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:05.989910 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 12:41:05.989958 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 12:41:05.990921 5332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:05.985864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46117 (local address 127.0.98.2:52645)
0927 12:41:05.986546 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:05.986596 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:05.988019 (+ 1423us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:05.988576 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:05.988585 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:05.988705 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:05.989322 (+ 617us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 12:41:05.989324 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:05.989366 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 12:41:05.989442 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:41:05.989772 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:41:05.989779 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:41:05.989784 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 12:41:05.989926 (+ 142us) client_negotiation.cc:240] Negotiation successful
0927 12:41:05.990110 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":128,"threads_started":1}
I20260927 12:41:05.992091 5330 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46117
I20260927 12:41:05.992319 5330 heartbeater.cc:464] Registering TS with master...
I20260927 12:41:05.992705 5330 heartbeater.cc:511] Master 127.0.98.62:46117 requested a full tablet report, sending...
I20260927 12:41:05.993769 5011 ts_manager.cc:194] Registered new tserver with Master: 8ce7cc6d39fa49e2b5ad2da45f45977a (127.0.98.2:42189)
W20260927 12:41:06.099556 5333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:41:06.099869 5333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:41:06.099954 5333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:41:06.104202 5333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:41:06.104259 5333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:41:06.104316 5333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:41:06.109157 5333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512840836255-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512840836255-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512840836255-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:41:06.110293 5333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:41:06.111644 5333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:41:06.120816 5341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:06.120851 5339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:06.120864 5338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:06.120985 5333 server_base.cc:1034] running on GCE node
I20260927 12:41:06.121505 5333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:41:06.122308 5333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:41:06.123509 5333 hybrid_clock.cc:648] HybridClock initialized: now 1790512866123489 us; error 36 us; skew 500 ppm
I20260927 12:41:06.125309 5333 webserver.cc:460] Webserver started at http://127.0.98.3:35145/ using document root <none> and password file <none>
I20260927 12:41:06.125998 5333 fs_manager.cc:359] Metadata directory not provided
I20260927 12:41:06.126088 5333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:41:06.126310 5333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:41:06.128284 5333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512840836255-392-0/minicluster-data/ts-2/data/instance:
uuid: "d46ddc6c144e4880aec7b96cf31780d3"
format_stamp: "Formatted at 2026-09-27 12:41:06 on dist-test-slave-bnc1"
server_key: "7f4525dbdd1f8022732d14e50e2df223"
server_key_iv: "a38f3c251d94e105f5479f2a675ec62a"
server_key_version: "encryptionkey@0"
I20260927 12:41:06.128844 5333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512840836255-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d46ddc6c144e4880aec7b96cf31780d3"
format_stamp: "Formatted at 2026-09-27 12:41:06 on dist-test-slave-bnc1"
server_key: "7f4525dbdd1f8022732d14e50e2df223"
server_key_iv: "a38f3c251d94e105f5479f2a675ec62a"
server_key_version: "encryptionkey@0"
I20260927 12:41:06.132356 5333 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 12:41:06.134738 5347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:06.135898 5333 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 12:41:06.136030 5333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512840836255-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512840836255-392-0/minicluster-data/ts-2/wal
uuid: "d46ddc6c144e4880aec7b96cf31780d3"
format_stamp: "Formatted at 2026-09-27 12:41:06 on dist-test-slave-bnc1"
server_key: "7f4525dbdd1f8022732d14e50e2df223"
server_key_iv: "a38f3c251d94e105f5479f2a675ec62a"
server_key_version: "encryptionkey@0"
I20260927 12:41:06.136145 5333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512840836255-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512840836255-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512840836255-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:41:06.140681 5333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:41:06.141268 5333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:41:06.141462 5333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:41:06.142045 5333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:41:06.143054 5333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:41:06.143123 5333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:06.143188 5333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:41:06.143236 5333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:06.151988 5333 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42891
I20260927 12:41:06.152009 5460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42891 every 8 connection(s)
I20260927 12:41:06.152870 5333 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:41:06.158674 5050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:06.154549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39187 (local address 127.0.98.62:46117)
0927 12:41:06.154678 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:06.154682 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:06.155743 (+ 1061us) server_negotiation.cc:407] Connection header received
0927 12:41:06.157001 (+ 1258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:06.157003 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:06.157037 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:06.157120 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:06.158279 (+ 1159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:06.158282 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:06.158284 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:41:06.158297 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:06.158316 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:41:06.158324 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:06.158412 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:06.158534 (+ 122us) server_negotiation.cc:299] Negotiation successful
0927 12:41:06.158571 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 12:41:06.159525 5463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:06.154917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46117 (local address 127.0.98.3:39187)
0927 12:41:06.155590 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:06.155638 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:06.156822 (+ 1184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:06.157271 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:06.157278 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:06.157364 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:06.157994 (+ 630us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 12:41:06.157995 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:06.158036 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 12:41:06.158103 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:41:06.158407 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:41:06.158413 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:41:06.158417 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 12:41:06.158548 (+ 131us) client_negotiation.cc:240] Negotiation successful
0927 12:41:06.158733 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":128,"threads_started":1}
I20260927 12:41:06.160589 5461 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46117
I20260927 12:41:06.160789 5461 heartbeater.cc:464] Registering TS with master...
I20260927 12:41:06.161167 5461 heartbeater.cc:511] Master 127.0.98.62:46117 requested a full tablet report, sending...
I20260927 12:41:06.162305 5011 ts_manager.cc:194] Registered new tserver with Master: d46ddc6c144e4880aec7b96cf31780d3 (127.0.98.3:42891)
I20260927 12:41:06.163143 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 5333
I20260927 12:41:06.163264 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512840836255-392-0/minicluster-data/ts-2/wal/instance
I20260927 12:41:06.163558 392 external_mini_cluster.cc:1428] Setting key 556f0ff1f735aa0859073ecf2407d809
I20260927 12:41:06.165477 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 12:41:06.168887 5050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:06.167304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59830 (local address 127.0.98.62:46117)
0927 12:41:06.167446 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:06.167451 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:06.167670 (+ 219us) server_negotiation.cc:407] Connection header received
0927 12:41:06.167759 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:06.167761 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:06.167792 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:06.167855 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:06.168568 (+ 713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:06.168571 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:06.168572 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:41:06.168581 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:06.168595 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:41:06.168602 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:06.168661 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:06.168706 (+ 45us) server_negotiation.cc:299] Negotiation successful
0927 12:41:06.168734 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 12:41:06.171682 5011 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59830:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 12:41:06.174510 5011 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:41:06.185721 5475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:06.182966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41277 (local address 127.0.0.1:37628)
0927 12:41:06.183718 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:06.183733 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:06.183808 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:06.184211 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:06.184218 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:06.184293 (+ 75us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:06.184889 (+ 596us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 12:41:06.184891 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:06.184909 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 12:41:06.184983 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:41:06.185311 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:41:06.185319 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:41:06.185324 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 12:41:06.185452 (+ 128us) client_negotiation.cc:240] Negotiation successful
0927 12:41:06.185529 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"thread_start_us":87,"threads_started":1}
I20260927 12:41:06.186314 5476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:06.183350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37628 (local address 127.0.98.1:41277)
0927 12:41:06.183826 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:06.183838 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:06.183862 (+ 24us) server_negotiation.cc:407] Connection header received
0927 12:41:06.183913 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:06.183918 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:06.184067 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:06.184210 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:06.185128 (+ 918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:06.185136 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:06.185153 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:41:06.185162 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:06.185197 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:41:06.185216 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:06.185326 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:06.185475 (+ 149us) server_negotiation.cc:299] Negotiation successful
0927 12:41:06.185543 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":109,"threads_started":1}
I20260927 12:41:06.186483 5474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:06.183270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42891 (local address 127.0.0.1:53286)
0927 12:41:06.184910 (+ 1640us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:06.184924 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:06.185001 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:06.185366 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:06.185369 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:06.185382 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:06.185847 (+ 465us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 12:41:06.185849 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:06.185859 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 12:41:06.185886 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:41:06.186222 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:41:06.186225 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:41:06.186227 (+ 2us) client_negotiation.cc:769] Sending connection context
0927 12:41:06.186307 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 12:41:06.186341 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1580,"thread_start_us":93,"threads_started":1}
I20260927 12:41:06.186586 5477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:06.183672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53286 (local address 127.0.98.3:42891)
0927 12:41:06.184079 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:06.184090 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:06.185048 (+ 958us) server_negotiation.cc:407] Connection header received
0927 12:41:06.185116 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:06.185120 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:06.185240 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:06.185382 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:06.186026 (+ 644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:06.186034 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:06.186045 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:41:06.186057 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:06.186104 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:41:06.186131 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:06.186231 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:06.186338 (+ 107us) server_negotiation.cc:299] Negotiation successful
0927 12:41:06.186418 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":144,"threads_started":1}
I20260927 12:41:06.186967 5472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:06.182608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42189 (local address 127.0.0.1:58806)
0927 12:41:06.183402 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:06.183451 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:06.183620 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:06.184693 (+ 1073us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:06.184698 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:06.184714 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:06.186288 (+ 1574us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 12:41:06.186292 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:06.186302 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 12:41:06.186325 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:41:06.186704 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:41:06.186706 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:41:06.186709 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 12:41:06.186806 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 12:41:06.186841 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":69,"threads_started":1}
I20260927 12:41:06.187057 5473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:06.182756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58806 (local address 127.0.98.2:42189)
0927 12:41:06.183147 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:06.183163 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:06.183544 (+ 381us) server_negotiation.cc:407] Connection header received
0927 12:41:06.183754 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:06.183761 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:06.183969 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:06.184098 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:06.186451 (+ 2353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:06.186456 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:06.186466 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:41:06.186478 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:06.186519 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:41:06.186557 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:06.186639 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:06.186836 (+ 197us) server_negotiation.cc:299] Negotiation successful
0927 12:41:06.186907 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":125,"threads_started":1}
I20260927 12:41:06.189335 5132 tablet_service.cc:1467] Processing CreateTablet for tablet 64e3fe1ea63547b39a74eaa9d7f69b2b (DEFAULT_TABLE table=test-table [id=344b42445b7e415bb97633c194a308b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:41:06.189363 5395 tablet_service.cc:1467] Processing CreateTablet for tablet 64e3fe1ea63547b39a74eaa9d7f69b2b (DEFAULT_TABLE table=test-table [id=344b42445b7e415bb97633c194a308b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:41:06.189492 5264 tablet_service.cc:1467] Processing CreateTablet for tablet 64e3fe1ea63547b39a74eaa9d7f69b2b (DEFAULT_TABLE table=test-table [id=344b42445b7e415bb97633c194a308b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:41:06.190310 5132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64e3fe1ea63547b39a74eaa9d7f69b2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:41:06.190338 5264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64e3fe1ea63547b39a74eaa9d7f69b2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:41:06.190338 5395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64e3fe1ea63547b39a74eaa9d7f69b2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:41:06.195912 5478 tablet_bootstrap.cc:492] T 64e3fe1ea63547b39a74eaa9d7f69b2b P a4d6279803074fb290c0c0e298a275ea: Bootstrap starting.
I20260927 12:41:06.197420 5479 tablet_bootstrap.cc:492] T 64e3fe1ea63547b39a74eaa9d7f69b2b P d46ddc6c144e4880aec7b96cf31780d3: Bootstrap starting.
I20260927 12:41:06.197773 5480 tablet_bootstrap.cc:492] T 64e3fe1ea63547b39a74eaa9d7f69b2b P 8ce7cc6d39fa49e2b5ad2da45f45977a: Bootstrap starting.
I20260927 12:41:06.198443 5478 tablet_bootstrap.cc:654] T 64e3fe1ea63547b39a74eaa9d7f69b2b P a4d6279803074fb290c0c0e298a275ea: Neither blocks nor log segments found. Creating new log.
I20260927 12:41:06.199249 5478 log.cc:824] T 64e3fe1ea63547b39a74eaa9d7f69b2b P a4d6279803074fb290c0c0e298a275ea: Log is configured to *not* fsync() on all Append() calls
I20260927 12:41:06.199520 5480 tablet_bootstrap.cc:654] T 64e3fe1ea63547b39a74eaa9d7f69b2b P 8ce7cc6d39fa49e2b5ad2da45f45977a: Neither blocks nor log segments found. Creating new log.
I20260927 12:41:06.200286 5480 log.cc:824] T 64e3fe1ea63547b39a74eaa9d7f69b2b P 8ce7cc6d39fa49e2b5ad2da45f45977a: Log is configured to *not* fsync() on all Append() calls
I20260927 12:41:06.201231 5478 tablet_bootstrap.cc:492] T 64e3fe1ea63547b39a74eaa9d7f69b2b P a4d6279803074fb290c0c0e298a275ea: No bootstrap required, opened a new log
I20260927 12:41:06.201409 5478 ts_tablet_manager.cc:1397] T 64e3fe1ea63547b39a74eaa9d7f69b2b P a4d6279803074fb290c0c0e298a275ea: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 12:41:06.201828 5479 tablet_bootstrap.cc:654] T 64e3fe1ea63547b39a74eaa9d7f69b2b P d46ddc6c144e4880aec7b96cf31780d3: Neither blocks nor log segments found. Creating new log.
I20260927 12:41:06.202203 5480 tablet_bootstrap.cc:492] T 64e3fe1ea63547b39a74eaa9d7f69b2b P 8ce7cc6d39fa49e2b5ad2da45f45977a: No bootstrap required, opened a new log
I20260927 12:41:06.202432 5480 ts_tablet_manager.cc:1397] T 64e3fe1ea63547b39a74eaa9d7f69b2b P 8ce7cc6d39fa49e2b5ad2da45f45977a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260927 12:41:06.202646 5479 log.cc:824] T 64e3fe1ea63547b39a74eaa9d7f69b2b P d46ddc6c144e4880aec7b96cf31780d3: Log is configured to *not* fsync() on all Append() calls
I20260927 12:41:06.204165 5479 tablet_bootstrap.cc:492] T 64e3fe1ea63547b39a74eaa9d7f69b2b P d46ddc6c144e4880aec7b96cf31780d3: No bootstrap required, opened a new log
I20260927 12:41:06.204355 5479 ts_tablet_manager.cc:1397] T 64e3fe1ea63547b39a74eaa9d7f69b2b P d46ddc6c144e4880aec7b96cf31780d3: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260927 12:41:06.205988 5478 raft_consensus.cc:348] T 64e3fe1ea63547b39a74eaa9d7f69b2b P a4d6279803074fb290c0c0e298a275ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ce7cc6d39fa49e2b5ad2da45f45977a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42189 } } peers { permanent_uuid: "d46ddc6c144e4880aec7b96cf31780d3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42891 } } peers { permanent_uuid: "a4d6279803074fb290c0c0e298a275ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41277 } }
I20260927 12:41:06.206249 5478 raft_consensus.cc:374] T 64e3fe1ea63547b39a74eaa9d7f69b2b P a4d6279803074fb290c0c0e298a275ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:41:06.206339 5478 raft_consensus.cc:729] T 64e3fe1ea63547b39a74eaa9d7f69b2b P a4d6279803074fb290c0c0e298a275ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4d6279803074fb290c0c0e298a275ea, State: Initialized, Role: FOLLOWER
I20260927 12:41:06.206787 5478 consensus_queue.cc:260] T 64e3fe1ea63547b39a74eaa9d7f69b2b P a4d6279803074fb290c0c0e298a275ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ce7cc6d39fa49e2b5ad2da45f45977a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42189 } } peers { permanent_uuid: "d46ddc6c144e4880aec7b96cf31780d3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42891 } } peers { permanent_uuid: "a4d6279803074fb290c0c0e298a275ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41277 } }
I20260927 12:41:06.206823 5480 raft_consensus.cc:348] T 64e3fe1ea63547b39a74eaa9d7f69b2b P 8ce7cc6d39fa49e2b5ad2da45f45977a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ce7cc6d39fa49e2b5ad2da45f45977a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42189 } } peers { permanent_uuid: "d46ddc6c144e4880aec7b96cf31780d3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42891 } } peers { permanent_uuid: "a4d6279803074fb290c0c0e298a275ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41277 } }
I20260927 12:41:06.207103 5480 raft_consensus.cc:374] T 64e3fe1ea63547b39a74eaa9d7f69b2b P 8ce7cc6d39fa49e2b5ad2da45f45977a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:41:06.207170 5480 raft_consensus.cc:729] T 64e3fe1ea63547b39a74eaa9d7f69b2b P 8ce7cc6d39fa49e2b5ad2da45f45977a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ce7cc6d39fa49e2b5ad2da45f45977a, State: Initialized, Role: FOLLOWER
I20260927 12:41:06.207548 5480 consensus_queue.cc:260] T 64e3fe1ea63547b39a74eaa9d7f69b2b P 8ce7cc6d39fa49e2b5ad2da45f45977a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ce7cc6d39fa49e2b5ad2da45f45977a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42189 } } peers { permanent_uuid: "d46ddc6c144e4880aec7b96cf31780d3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42891 } } peers { permanent_uuid: "a4d6279803074fb290c0c0e298a275ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41277 } }
I20260927 12:41:06.207785 5478 ts_tablet_manager.cc:1428] T 64e3fe1ea63547b39a74eaa9d7f69b2b P a4d6279803074fb290c0c0e298a275ea: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260927 12:41:06.208066 5479 raft_consensus.cc:348] T 64e3fe1ea63547b39a74eaa9d7f69b2b P d46ddc6c144e4880aec7b96cf31780d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ce7cc6d39fa49e2b5ad2da45f45977a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42189 } } peers { permanent_uuid: "d46ddc6c144e4880aec7b96cf31780d3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42891 } } peers { permanent_uuid: "a4d6279803074fb290c0c0e298a275ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41277 } }
I20260927 12:41:06.208236 5199 heartbeater.cc:503] Master 127.0.98.62:46117 was elected leader, sending a full tablet report...
I20260927 12:41:06.208313 5479 raft_consensus.cc:374] T 64e3fe1ea63547b39a74eaa9d7f69b2b P d46ddc6c144e4880aec7b96cf31780d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:41:06.208401 5479 raft_consensus.cc:729] T 64e3fe1ea63547b39a74eaa9d7f69b2b P d46ddc6c144e4880aec7b96cf31780d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d46ddc6c144e4880aec7b96cf31780d3, State: Initialized, Role: FOLLOWER
I20260927 12:41:06.208618 5480 ts_tablet_manager.cc:1428] T 64e3fe1ea63547b39a74eaa9d7f69b2b P 8ce7cc6d39fa49e2b5ad2da45f45977a: Time spent starting tablet: real 0.006s user 0.004s sys 0.002s
I20260927 12:41:06.208846 5479 consensus_queue.cc:260] T 64e3fe1ea63547b39a74eaa9d7f69b2b P d46ddc6c144e4880aec7b96cf31780d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ce7cc6d39fa49e2b5ad2da45f45977a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42189 } } peers { permanent_uuid: "d46ddc6c144e4880aec7b96cf31780d3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42891 } } peers { permanent_uuid: "a4d6279803074fb290c0c0e298a275ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41277 } }
I20260927 12:41:06.208942 5330 heartbeater.cc:503] Master 127.0.98.62:46117 was elected leader, sending a full tablet report...
I20260927 12:41:06.209844 5461 heartbeater.cc:503] Master 127.0.98.62:46117 was elected leader, sending a full tablet report...
I20260927 12:41:06.209882 5479 ts_tablet_manager.cc:1428] T 64e3fe1ea63547b39a74eaa9d7f69b2b P d46ddc6c144e4880aec7b96cf31780d3: Time spent starting tablet: real 0.005s user 0.002s sys 0.005s
W20260927 12:41:06.234506 5331 tablet.cc:2369] T 64e3fe1ea63547b39a74eaa9d7f69b2b P 8ce7cc6d39fa49e2b5ad2da45f45977a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 12:41:06.312856 5200 tablet.cc:2369] T 64e3fe1ea63547b39a74eaa9d7f69b2b P a4d6279803074fb290c0c0e298a275ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 12:41:06.403837 5462 tablet.cc:2369] T 64e3fe1ea63547b39a74eaa9d7f69b2b P d46ddc6c144e4880aec7b96cf31780d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:41:06.493677 5484 raft_consensus.cc:482] T 64e3fe1ea63547b39a74eaa9d7f69b2b P a4d6279803074fb290c0c0e298a275ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:41:06.493880 5484 raft_consensus.cc:504] T 64e3fe1ea63547b39a74eaa9d7f69b2b P a4d6279803074fb290c0c0e298a275ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ce7cc6d39fa49e2b5ad2da45f45977a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42189 } } peers { permanent_uuid: "d46ddc6c144e4880aec7b96cf31780d3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42891 } } peers { permanent_uuid: "a4d6279803074fb290c0c0e298a275ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41277 } }
I20260927 12:41:06.495070 5484 leader_election.cc:288] T 64e3fe1ea63547b39a74eaa9d7f69b2b P a4d6279803074fb290c0c0e298a275ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ce7cc6d39fa49e2b5ad2da45f45977a (127.0.98.2:42189), d46ddc6c144e4880aec7b96cf31780d3 (127.0.98.3:42891)
I20260927 12:41:06.497036 5473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:06.495354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49677 (local address 127.0.98.2:42189)
0927 12:41:06.495491 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:06.495495 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:06.495646 (+ 151us) server_negotiation.cc:407] Connection header received
0927 12:41:06.495748 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:06.495754 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:06.495796 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:06.495863 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:06.496582 (+ 719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:06.496586 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:06.496589 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:41:06.496608 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:06.496626 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:41:06.496660 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:06.496779 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:06.496848 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 12:41:06.496893 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 12:41:06.497064 5487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:06.495251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42189 (local address 127.0.98.1:49677)
0927 12:41:06.495498 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:06.495513 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:06.495639 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:06.495911 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:06.495914 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:06.495927 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:06.496451 (+ 524us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 12:41:06.496453 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:06.496462 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 12:41:06.496485 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:41:06.496764 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:41:06.496766 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:41:06.496776 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 12:41:06.496835 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 12:41:06.496864 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":78,"threads_started":1}
I20260927 12:41:06.497162 5477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:06.495701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37215 (local address 127.0.98.3:42891)
0927 12:41:06.495823 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:06.495826 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:06.495837 (+ 11us) server_negotiation.cc:407] Connection header received
0927 12:41:06.495901 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:06.495904 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:06.495937 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:06.496020 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:06.496781 (+ 761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:06.496785 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:06.496790 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:41:06.496804 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:06.496826 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:41:06.496837 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:06.496922 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:06.496996 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 12:41:06.497027 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 12:41:06.497229 5488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:06.495321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42891 (local address 127.0.98.1:37215)
0927 12:41:06.495554 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:06.495569 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:06.495673 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:06.496051 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:06.496053 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:06.496065 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:06.496629 (+ 564us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 12:41:06.496631 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:06.496645 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 12:41:06.496663 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:41:06.496921 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:41:06.496924 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:41:06.496927 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 12:41:06.496997 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 12:41:06.497023 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":79,"threads_started":1}
I20260927 12:41:06.497917 5415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64e3fe1ea63547b39a74eaa9d7f69b2b" candidate_uuid: "a4d6279803074fb290c0c0e298a275ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d46ddc6c144e4880aec7b96cf31780d3" is_pre_election: true
I20260927 12:41:06.497910 5284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64e3fe1ea63547b39a74eaa9d7f69b2b" candidate_uuid: "a4d6279803074fb290c0c0e298a275ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ce7cc6d39fa49e2b5ad2da45f45977a" is_pre_election: true
I20260927 12:41:06.498215 5415 raft_consensus.cc:2454] T 64e3fe1ea63547b39a74eaa9d7f69b2b P d46ddc6c144e4880aec7b96cf31780d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4d6279803074fb290c0c0e298a275ea in term 0.
I20260927 12:41:06.498215 5284 raft_consensus.cc:2454] T 64e3fe1ea63547b39a74eaa9d7f69b2b P 8ce7cc6d39fa49e2b5ad2da45f45977a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4d6279803074fb290c0c0e298a275ea in term 0.
I20260927 12:41:06.498692 5087 leader_election.cc:302] T 64e3fe1ea63547b39a74eaa9d7f69b2b P a4d6279803074fb290c0c0e298a275ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ce7cc6d39fa49e2b5ad2da45f45977a, a4d6279803074fb290c0c0e298a275ea; no voters:
I20260927 12:41:06.498950 5484 raft_consensus.cc:2781] T 64e3fe1ea63547b39a74eaa9d7f69b2b P a4d6279803074fb290c0c0e298a275ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:41:06.499032 5484 raft_consensus.cc:482] T 64e3fe1ea63547b39a74eaa9d7f69b2b P a4d6279803074fb290c0c0e298a275ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:41:06.499089 5484 raft_consensus.cc:3037] T 64e3fe1ea63547b39a74eaa9d7f69b2b P a4d6279803074fb290c0c0e298a275ea [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:41:06.500192 5484 raft_consensus.cc:504] T 64e3fe1ea63547b39a74eaa9d7f69b2b P a4d6279803074fb290c0c0e298a275ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ce7cc6d39fa49e2b5ad2da45f45977a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42189 } } peers { permanent_uuid: "d46ddc6c144e4880aec7b96cf31780d3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42891 } } peers { permanent_uuid: "a4d6279803074fb290c0c0e298a275ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41277 } }
I20260927 12:41:06.500540 5484 leader_election.cc:288] T 64e3fe1ea63547b39a74eaa9d7f69b2b P a4d6279803074fb290c0c0e298a275ea [CANDIDATE]: Term 1 election: Requested vote from peers 8ce7cc6d39fa49e2b5ad2da45f45977a (127.0.98.2:42189), d46ddc6c144e4880aec7b96cf31780d3 (127.0.98.3:42891)
I20260927 12:41:06.500862 5415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64e3fe1ea63547b39a74eaa9d7f69b2b" candidate_uuid: "a4d6279803074fb290c0c0e298a275ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d46ddc6c144e4880aec7b96cf31780d3"
I20260927 12:41:06.500873 5284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64e3fe1ea63547b39a74eaa9d7f69b2b" candidate_uuid: "a4d6279803074fb290c0c0e298a275ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ce7cc6d39fa49e2b5ad2da45f45977a"
I20260927 12:41:06.501004 5284 raft_consensus.cc:3037] T 64e3fe1ea63547b39a74eaa9d7f69b2b P 8ce7cc6d39fa49e2b5ad2da45f45977a [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:41:06.501016 5415 raft_consensus.cc:3037] T 64e3fe1ea63547b39a74eaa9d7f69b2b P d46ddc6c144e4880aec7b96cf31780d3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:41:06.502115 5415 raft_consensus.cc:2454] T 64e3fe1ea63547b39a74eaa9d7f69b2b P d46ddc6c144e4880aec7b96cf31780d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4d6279803074fb290c0c0e298a275ea in term 1.
I20260927 12:41:06.502112 5284 raft_consensus.cc:2454] T 64e3fe1ea63547b39a74eaa9d7f69b2b P 8ce7cc6d39fa49e2b5ad2da45f45977a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4d6279803074fb290c0c0e298a275ea in term 1.
I20260927 12:41:06.502427 5086 leader_election.cc:302] T 64e3fe1ea63547b39a74eaa9d7f69b2b P a4d6279803074fb290c0c0e298a275ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4d6279803074fb290c0c0e298a275ea, d46ddc6c144e4880aec7b96cf31780d3; no voters:
I20260927 12:41:06.502665 5484 raft_consensus.cc:2781] T 64e3fe1ea63547b39a74eaa9d7f69b2b P a4d6279803074fb290c0c0e298a275ea [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:41:06.502907 5484 raft_consensus.cc:686] T 64e3fe1ea63547b39a74eaa9d7f69b2b P a4d6279803074fb290c0c0e298a275ea [term 1 LEADER]: Becoming Leader. State: Replica: a4d6279803074fb290c0c0e298a275ea, State: Running, Role: LEADER
I20260927 12:41:06.503242 5484 consensus_queue.cc:237] T 64e3fe1ea63547b39a74eaa9d7f69b2b P a4d6279803074fb290c0c0e298a275ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ce7cc6d39fa49e2b5ad2da45f45977a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42189 } } peers { permanent_uuid: "d46ddc6c144e4880aec7b96cf31780d3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42891 } } peers { permanent_uuid: "a4d6279803074fb290c0c0e298a275ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41277 } }
I20260927 12:41:06.506407 5011 catalog_manager.cc:5495] T 64e3fe1ea63547b39a74eaa9d7f69b2b P a4d6279803074fb290c0c0e298a275ea reported cstate change: term changed from 0 to 1, leader changed from <none> to a4d6279803074fb290c0c0e298a275ea (127.0.98.1). New cstate: current_term: 1 leader_uuid: "a4d6279803074fb290c0c0e298a275ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ce7cc6d39fa49e2b5ad2da45f45977a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d46ddc6c144e4880aec7b96cf31780d3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4d6279803074fb290c0c0e298a275ea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41277 } health_report { overall_health: HEALTHY } } }
I20260927 12:41:06.553187 5476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:06.551752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37636 (local address 127.0.98.1:41277)
0927 12:41:06.551883 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:06.551889 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:06.551908 (+ 19us) server_negotiation.cc:407] Connection header received
0927 12:41:06.551999 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:06.552001 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:06.552060 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:06.552153 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:06.552755 (+ 602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:06.552758 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:06.552761 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:41:06.552772 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:06.552799 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:41:06.552807 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:06.552898 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:06.553006 (+ 108us) server_negotiation.cc:299] Negotiation successful
0927 12:41:06.553044 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 12:41:06.559216 5284 raft_consensus.cc:1261] T 64e3fe1ea63547b39a74eaa9d7f69b2b P 8ce7cc6d39fa49e2b5ad2da45f45977a [term 1 FOLLOWER]: Refusing update from remote peer a4d6279803074fb290c0c0e298a275ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:41:06.559268 5415 raft_consensus.cc:1261] T 64e3fe1ea63547b39a74eaa9d7f69b2b P d46ddc6c144e4880aec7b96cf31780d3 [term 1 FOLLOWER]: Refusing update from remote peer a4d6279803074fb290c0c0e298a275ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:41:06.559969 5489 consensus_queue.cc:1035] T 64e3fe1ea63547b39a74eaa9d7f69b2b P a4d6279803074fb290c0c0e298a275ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "d46ddc6c144e4880aec7b96cf31780d3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:41:06.560155 5484 consensus_queue.cc:1035] T 64e3fe1ea63547b39a74eaa9d7f69b2b P a4d6279803074fb290c0c0e298a275ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ce7cc6d39fa49e2b5ad2da45f45977a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:41:06.569330 5494 mvcc.cc:205] Tried to move back new op lower bound from 7333940701418205184 to 7333940701199331328. Current Snapshot: MvccSnapshot[applied={T|T < 7333940701418205184}]
I20260927 12:41:06.570333 5493 mvcc.cc:205] Tried to move back new op lower bound from 7333940701418205184 to 7333940701199331328. Current Snapshot: MvccSnapshot[applied={T|T < 7333940701418205184}]
I20260927 12:41:06.575147 5011 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:41:06.575325 5011 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:41:06.577725 5011 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 04bcfe09cab24583aad7cd6c4e551461: Sending DeleteTablet for 3 replicas of tablet 64e3fe1ea63547b39a74eaa9d7f69b2b
I20260927 12:41:06.578620 5264 tablet_service.cc:1514] Processing DeleteTablet for tablet 64e3fe1ea63547b39a74eaa9d7f69b2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:41:06 UTC) from {username='slave'} at 127.0.0.1:58806
I20260927 12:41:06.578619 5132 tablet_service.cc:1514] Processing DeleteTablet for tablet 64e3fe1ea63547b39a74eaa9d7f69b2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:41:06 UTC) from {username='slave'} at 127.0.0.1:37628
I20260927 12:41:06.578619 5395 tablet_service.cc:1514] Processing DeleteTablet for tablet 64e3fe1ea63547b39a74eaa9d7f69b2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:41:06 UTC) from {username='slave'} at 127.0.0.1:53286
I20260927 12:41:06.579129 5504 tablet_replica.cc:330] T 64e3fe1ea63547b39a74eaa9d7f69b2b P 8ce7cc6d39fa49e2b5ad2da45f45977a: stopping tablet replica
I20260927 12:41:06.579425 5504 raft_consensus.cc:2229] T 64e3fe1ea63547b39a74eaa9d7f69b2b P 8ce7cc6d39fa49e2b5ad2da45f45977a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:41:06.579649 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 5071
I20260927 12:41:06.579751 5504 raft_consensus.cc:2258] T 64e3fe1ea63547b39a74eaa9d7f69b2b P 8ce7cc6d39fa49e2b5ad2da45f45977a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:41:06.580521 5503 tablet_replica.cc:330] T 64e3fe1ea63547b39a74eaa9d7f69b2b P d46ddc6c144e4880aec7b96cf31780d3: stopping tablet replica
I20260927 12:41:06.580823 5503 raft_consensus.cc:2229] T 64e3fe1ea63547b39a74eaa9d7f69b2b P d46ddc6c144e4880aec7b96cf31780d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:41:06.581125 5503 raft_consensus.cc:2258] T 64e3fe1ea63547b39a74eaa9d7f69b2b P d46ddc6c144e4880aec7b96cf31780d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:41:06.581990 5504 ts_tablet_manager.cc:1905] T 64e3fe1ea63547b39a74eaa9d7f69b2b P 8ce7cc6d39fa49e2b5ad2da45f45977a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:41:06.582543 5503 ts_tablet_manager.cc:1905] T 64e3fe1ea63547b39a74eaa9d7f69b2b P d46ddc6c144e4880aec7b96cf31780d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:41:06.584345 5504 ts_tablet_manager.cc:1918] T 64e3fe1ea63547b39a74eaa9d7f69b2b P 8ce7cc6d39fa49e2b5ad2da45f45977a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:41:06.584457 5504 log.cc:1196] T 64e3fe1ea63547b39a74eaa9d7f69b2b P 8ce7cc6d39fa49e2b5ad2da45f45977a: Deleting WAL directory at /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512840836255-392-0/minicluster-data/ts-1/wal/wals/64e3fe1ea63547b39a74eaa9d7f69b2b
I20260927 12:41:06.584751 5504 ts_tablet_manager.cc:1939] T 64e3fe1ea63547b39a74eaa9d7f69b2b P 8ce7cc6d39fa49e2b5ad2da45f45977a: Deleting consensus metadata
I20260927 12:41:06.586036 5503 ts_tablet_manager.cc:1918] T 64e3fe1ea63547b39a74eaa9d7f69b2b P d46ddc6c144e4880aec7b96cf31780d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:41:06.586131 5503 log.cc:1196] T 64e3fe1ea63547b39a74eaa9d7f69b2b P d46ddc6c144e4880aec7b96cf31780d3: Deleting WAL directory at /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790512840836255-392-0/minicluster-data/ts-2/wal/wals/64e3fe1ea63547b39a74eaa9d7f69b2b
I20260927 12:41:06.586114 4996 catalog_manager.cc:4841] TS 8ce7cc6d39fa49e2b5ad2da45f45977a (127.0.98.2:42189): tablet 64e3fe1ea63547b39a74eaa9d7f69b2b (table test-table [id=344b42445b7e415bb97633c194a308b7]) successfully deleted
I20260927 12:41:06.586431 5503 ts_tablet_manager.cc:1939] T 64e3fe1ea63547b39a74eaa9d7f69b2b P d46ddc6c144e4880aec7b96cf31780d3: Deleting consensus metadata
I20260927 12:41:06.587652 4995 catalog_manager.cc:4841] TS d46ddc6c144e4880aec7b96cf31780d3 (127.0.98.3:42891): tablet 64e3fe1ea63547b39a74eaa9d7f69b2b (table test-table [id=344b42445b7e415bb97633c194a308b7]) successfully deleted
I20260927 12:41:06.588224 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 5202
W20260927 12:41:06.588425 4998 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41277 (error 108)
I20260927 12:41:06.589689 5474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:06.589341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41277 (local address 127.0.0.1:37640)
0927 12:41:06.589492 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:06.589591 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 12:41:06.590342 4998 catalog_manager.cc:4584] TS a4d6279803074fb290c0c0e298a275ea (127.0.98.1:41277): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 64e3fe1ea63547b39a74eaa9d7f69b2b: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41277: connect: Connection refused (error 111)
I20260927 12:41:06.595691 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 5333
I20260927 12:41:06.601168 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 4980
2026-09-27T12:41:06Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1200 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260927 12:41:06.623257 392 mini_oidc.cc:152] Starting JWKS server
I20260927 12:41:06.625427 392 webserver.cc:460] Webserver started at https://127.0.0.1:43491/ using document root <none> and password file <none>
I20260927 12:41:06.625633 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 12:41:06.625914 392 webserver.cc:460] Webserver started at http://127.0.0.1:40755/ using document root <none> and password file <none>
2026-09-27T12:41:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:41:06Z Disabled control of system clock
I20260927 12:41:06.634773 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790512840836255-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790512840836255-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790512840836255-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43661
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43661
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43491/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790512840836255-392-0/testchainca.cert with env {}
W20260927 12:41:06.743646 5511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:41:06.743903 5511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:41:06.743942 5511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:41:06.747960 5511 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:41:06.748024 5511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:41:06.748039 5511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:41:06.748054 5511 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:41:06.748070 5511 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 12:41:06.748085 5511 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 12:41:06.748102 5511 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43491/jwks.json
I20260927 12:41:06.753238 5511 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790512840836255-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790512840836255-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43661
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43491/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790512840836255-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5511
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790512840836255-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:41:06.754176 5511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:41:06.755093 5511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:41:06.763370 5519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:06.763399 5517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:06.763427 5516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:06.763715 5511 server_base.cc:1034] running on GCE node
I20260927 12:41:06.764245 5511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:41:06.765290 5511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:41:06.766451 5511 hybrid_clock.cc:648] HybridClock initialized: now 1790512866766452 us; error 73 us; skew 500 ppm
I20260927 12:41:06.768213 5511 webserver.cc:460] Webserver started at http://127.0.98.62:45251/ using document root <none> and password file <none>
I20260927 12:41:06.768777 5511 fs_manager.cc:359] Metadata directory not provided
I20260927 12:41:06.768836 5511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:41:06.769063 5511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:41:06.771086 5511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790512840836255-392-0/minicluster-data/master-0/data/instance:
uuid: "6a760d6dae49418fa0bcd3d4bd83a019"
format_stamp: "Formatted at 2026-09-27 12:41:06 on dist-test-slave-bnc1"
server_key: "07d9d2fb9f895c70854ea74b79b35e7a"
server_key_iv: "43cf45e6bb5b3c5d4b1dd74ea93d3e46"
server_key_version: "encryptionkey@0"
I20260927 12:41:06.771622 5511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790512840836255-392-0/minicluster-data/master-0/wal/instance:
uuid: "6a760d6dae49418fa0bcd3d4bd83a019"
format_stamp: "Formatted at 2026-09-27 12:41:06 on dist-test-slave-bnc1"
server_key: "07d9d2fb9f895c70854ea74b79b35e7a"
server_key_iv: "43cf45e6bb5b3c5d4b1dd74ea93d3e46"
server_key_version: "encryptionkey@0"
I20260927 12:41:06.775038 5511 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260927 12:41:06.777257 5525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:06.778474 5511 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:41:06.778605 5511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790512840836255-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790512840836255-392-0/minicluster-data/master-0/wal
uuid: "6a760d6dae49418fa0bcd3d4bd83a019"
format_stamp: "Formatted at 2026-09-27 12:41:06 on dist-test-slave-bnc1"
server_key: "07d9d2fb9f895c70854ea74b79b35e7a"
server_key_iv: "43cf45e6bb5b3c5d4b1dd74ea93d3e46"
server_key_version: "encryptionkey@0"
I20260927 12:41:06.778715 5511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790512840836255-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790512840836255-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790512840836255-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:41:06.790514 5511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:41:06.791129 5511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:41:06.791316 5511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:41:06.798475 5511 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43661
I20260927 12:41:06.798484 5577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43661 every 8 connection(s)
I20260927 12:41:06.799356 5511 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:41:06.799978 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 5511
I20260927 12:41:06.800102 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790512840836255-392-0/minicluster-data/master-0/wal/instance
I20260927 12:41:06.800391 392 external_mini_cluster.cc:1428] Setting key 2df3f8d1b5a3765aaf648d6153997450
I20260927 12:41:06.802842 5578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:41:06.808126 5578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a760d6dae49418fa0bcd3d4bd83a019: Bootstrap starting.
I20260927 12:41:06.809996 5581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:06.801866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48388 (local address 127.0.98.62:43661)
0927 12:41:06.803067 (+ 1201us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:06.803086 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:06.803120 (+ 34us) server_negotiation.cc:407] Connection header received
0927 12:41:06.803745 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:06.803760 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:06.804124 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:06.804469 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:06.804864 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:06.805866 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:06.806755 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:06.807221 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:06.807394 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:06.808230 (+ 836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:06.808237 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:06.808252 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:41:06.808647 (+ 395us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:06.808700 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:41:06.808720 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:06.808827 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:06.808991 (+ 164us) server_negotiation.cc:299] Negotiation successful
0927 12:41:06.809241 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":833,"thread_start_us":683,"threads_started":1}
I20260927 12:41:06.810410 5578 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a760d6dae49418fa0bcd3d4bd83a019: Neither blocks nor log segments found. Creating new log.
I20260927 12:41:06.811203 5578 log.cc:824] T 00000000000000000000000000000000 P 6a760d6dae49418fa0bcd3d4bd83a019: Log is configured to *not* fsync() on all Append() calls
I20260927 12:41:06.812844 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 12:41:06.813321 5578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a760d6dae49418fa0bcd3d4bd83a019: No bootstrap required, opened a new log
I20260927 12:41:06.816890 5578 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a760d6dae49418fa0bcd3d4bd83a019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a760d6dae49418fa0bcd3d4bd83a019" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43661 } }
I20260927 12:41:06.817085 5578 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a760d6dae49418fa0bcd3d4bd83a019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:41:06.817118 5578 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a760d6dae49418fa0bcd3d4bd83a019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a760d6dae49418fa0bcd3d4bd83a019, State: Initialized, Role: FOLLOWER
I20260927 12:41:06.817456 5578 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a760d6dae49418fa0bcd3d4bd83a019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a760d6dae49418fa0bcd3d4bd83a019" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43661 } }
I20260927 12:41:06.817556 5578 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a760d6dae49418fa0bcd3d4bd83a019 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:41:06.817664 5578 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a760d6dae49418fa0bcd3d4bd83a019 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:41:06.817731 5578 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a760d6dae49418fa0bcd3d4bd83a019 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:41:06.818498 5578 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a760d6dae49418fa0bcd3d4bd83a019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a760d6dae49418fa0bcd3d4bd83a019" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43661 } }
I20260927 12:41:06.818770 5578 leader_election.cc:302] T 00000000000000000000000000000000 P 6a760d6dae49418fa0bcd3d4bd83a019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a760d6dae49418fa0bcd3d4bd83a019; no voters:
I20260927 12:41:06.818994 5578 leader_election.cc:288] T 00000000000000000000000000000000 P 6a760d6dae49418fa0bcd3d4bd83a019 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:41:06.819111 5583 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a760d6dae49418fa0bcd3d4bd83a019 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:41:06.819362 5583 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a760d6dae49418fa0bcd3d4bd83a019 [term 1 LEADER]: Becoming Leader. State: Replica: 6a760d6dae49418fa0bcd3d4bd83a019, State: Running, Role: LEADER
I20260927 12:41:06.819660 5583 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a760d6dae49418fa0bcd3d4bd83a019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a760d6dae49418fa0bcd3d4bd83a019" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43661 } }
I20260927 12:41:06.820179 5578 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a760d6dae49418fa0bcd3d4bd83a019 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:41:06.821501 5584 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a760d6dae49418fa0bcd3d4bd83a019 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a760d6dae49418fa0bcd3d4bd83a019" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a760d6dae49418fa0bcd3d4bd83a019" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43661 } } }
I20260927 12:41:06.821511 5585 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a760d6dae49418fa0bcd3d4bd83a019 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a760d6dae49418fa0bcd3d4bd83a019. Latest consensus state: current_term: 1 leader_uuid: "6a760d6dae49418fa0bcd3d4bd83a019" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a760d6dae49418fa0bcd3d4bd83a019" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43661 } } }
I20260927 12:41:06.821662 5584 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a760d6dae49418fa0bcd3d4bd83a019 [sys.catalog]: This master's current role is: LEADER
I20260927 12:41:06.821666 5585 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a760d6dae49418fa0bcd3d4bd83a019 [sys.catalog]: This master's current role is: LEADER
I20260927 12:41:06.822028 5592 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:41:06.825007 5592 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:41:06.830209 5592 catalog_manager.cc:1347] Generated new cluster ID: d8610c2a761544718d0311f7b501b3dd
I20260927 12:41:06.830269 5592 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:41:06.841373 5592 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:41:06.842258 5592 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:41:06.855492 5592 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a760d6dae49418fa0bcd3d4bd83a019: Generated new TSK 0
I20260927 12:41:06.856426 5592 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 12:41:06.896472 5608 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260927 12:41:06.896605 5608 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 12:41:06.897245 5581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:06.873982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48402 (local address 127.0.98.62:43661)
0927 12:41:06.874149 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:06.874152 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:06.874261 (+ 109us) server_negotiation.cc:407] Connection header received
0927 12:41:06.874471 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:06.874473 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:06.874517 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:06.874600 (+ 83us) server_negotiation.cc:226] Negotiated authn=JWT
0927 12:41:06.874881 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:06.875388 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:06.876597 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:06.876796 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:06.876864 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:06.877221 (+ 357us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 12:41:06.896762 (+ 19541us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43491/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 12:41:06.897086 (+ 324us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48402: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43491/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":99}
W20260927 12:41:06.897321 5581 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48402: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43491/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260927 12:41:06.898453 5607 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:41:06.873813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43661 (local address 127.0.0.1:48402)
0927 12:41:06.874174 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:06.874189 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:06.874315 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:06.874637 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:06.874640 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:06.874648 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0927 12:41:06.874777 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:06.874783 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:06.875511 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:06.875515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:06.876475 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:06.876488 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:06.876883 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:06.876902 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:06.876996 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:06.877047 (+ 51us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 12:41:06.897017 (+ 19970us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43491/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 12:41:06.897258 (+ 241us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:43661: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43491/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":119,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:43661: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43491/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260927 12:41:06.899142 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 5511
2026-09-27T12:41:06Z chronyd exiting
I20260927 12:41:06.937330 392 test_util.cc:175] -----------------------------------------------
I20260927 12:41:06.937419 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790512840836255-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (315 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260927 12:41:06.938503 392 mini_oidc.cc:152] Starting JWKS server
I20260927 12:41:06.939316 392 webserver.cc:460] Webserver started at https://127.0.0.1:40239/ using document root <none> and password file <none>
I20260927 12:41:06.939394 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 12:41:06.939601 392 webserver.cc:460] Webserver started at http://127.0.0.1:34301/ using document root <none> and password file <none>
2026-09-27T12:41:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:41:06Z Disabled control of system clock
I20260927 12:41:06.948292 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790512840836255-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790512840836255-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790512840836255-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45049
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:44609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45049
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40239/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790512840836255-392-0/testchainca.cert with env {}
W20260927 12:41:07.056834 5614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:41:07.057065 5614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:41:07.057101 5614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:41:07.060961 5614 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:41:07.061014 5614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:41:07.061029 5614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:41:07.061045 5614 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:41:07.061059 5614 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 12:41:07.061074 5614 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 12:41:07.061088 5614 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40239/jwks.json
I20260927 12:41:07.066339 5614 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790512840836255-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790512840836255-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45049
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40239/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790512840836255-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5614
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790512840836255-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:41:07.067286 5614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:41:07.068212 5614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:41:07.076689 5619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:07.076800 5614 server_base.cc:1034] running on GCE node
W20260927 12:41:07.076658 5620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:07.076663 5622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:07.077494 5614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:41:07.078560 5614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:41:07.079784 5614 hybrid_clock.cc:648] HybridClock initialized: now 1790512867079767 us; error 26 us; skew 500 ppm
I20260927 12:41:07.081806 5614 webserver.cc:460] Webserver started at http://127.0.98.62:40811/ using document root <none> and password file <none>
I20260927 12:41:07.082392 5614 fs_manager.cc:359] Metadata directory not provided
I20260927 12:41:07.082485 5614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:41:07.082727 5614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:41:07.084818 5614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790512840836255-392-0/minicluster-data/master-0/data/instance:
uuid: "d7726590f3744e87a84fe02c73e404e1"
format_stamp: "Formatted at 2026-09-27 12:41:07 on dist-test-slave-bnc1"
server_key: "b5b195fea220943749c2826875048e2b"
server_key_iv: "f68addc5a2b3329989173d1019a7957f"
server_key_version: "encryptionkey@0"
I20260927 12:41:07.085417 5614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790512840836255-392-0/minicluster-data/master-0/wal/instance:
uuid: "d7726590f3744e87a84fe02c73e404e1"
format_stamp: "Formatted at 2026-09-27 12:41:07 on dist-test-slave-bnc1"
server_key: "b5b195fea220943749c2826875048e2b"
server_key_iv: "f68addc5a2b3329989173d1019a7957f"
server_key_version: "encryptionkey@0"
I20260927 12:41:07.089038 5614 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 12:41:07.091439 5628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:07.092640 5614 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:41:07.092787 5614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790512840836255-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790512840836255-392-0/minicluster-data/master-0/wal
uuid: "d7726590f3744e87a84fe02c73e404e1"
format_stamp: "Formatted at 2026-09-27 12:41:07 on dist-test-slave-bnc1"
server_key: "b5b195fea220943749c2826875048e2b"
server_key_iv: "f68addc5a2b3329989173d1019a7957f"
server_key_version: "encryptionkey@0"
I20260927 12:41:07.092937 5614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790512840836255-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790512840836255-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790512840836255-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:41:07.110245 5614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:41:07.111040 5614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:41:07.111258 5614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:41:07.118647 5614 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45049
I20260927 12:41:07.118645 5680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45049 every 8 connection(s)
I20260927 12:41:07.119659 5614 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:41:07.122701 5681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:41:07.123942 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 5614
I20260927 12:41:07.124033 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790512840836255-392-0/minicluster-data/master-0/wal/instance
I20260927 12:41:07.124336 392 external_mini_cluster.cc:1428] Setting key 9f9bbfd4880abe1d63e8a8425f2ea401
I20260927 12:41:07.129135 5681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7726590f3744e87a84fe02c73e404e1: Bootstrap starting.
I20260927 12:41:07.131563 5681 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7726590f3744e87a84fe02c73e404e1: Neither blocks nor log segments found. Creating new log.
I20260927 12:41:07.132368 5681 log.cc:824] T 00000000000000000000000000000000 P d7726590f3744e87a84fe02c73e404e1: Log is configured to *not* fsync() on all Append() calls
I20260927 12:41:07.134549 5681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7726590f3744e87a84fe02c73e404e1: No bootstrap required, opened a new log
I20260927 12:41:07.134755 5684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:07.125784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53430 (local address 127.0.98.62:45049)
0927 12:41:07.126475 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:07.126494 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:07.126535 (+ 41us) server_negotiation.cc:407] Connection header received
0927 12:41:07.127377 (+ 842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:07.127400 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:07.127877 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:07.128255 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:07.128677 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:07.130097 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:07.131071 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:07.131761 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:07.131924 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:07.132745 (+ 821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:07.132753 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:07.132770 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:41:07.133207 (+ 437us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:07.133275 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:41:07.133300 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:07.133398 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:07.133599 (+ 201us) server_negotiation.cc:299] Negotiation successful
0927 12:41:07.133856 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":109,"threads_started":1}
I20260927 12:41:07.137552 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 12:41:07.138610 5681 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7726590f3744e87a84fe02c73e404e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7726590f3744e87a84fe02c73e404e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45049 } }
I20260927 12:41:07.138876 5681 raft_consensus.cc:374] T 00000000000000000000000000000000 P d7726590f3744e87a84fe02c73e404e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:41:07.138950 5681 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7726590f3744e87a84fe02c73e404e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7726590f3744e87a84fe02c73e404e1, State: Initialized, Role: FOLLOWER
I20260927 12:41:07.139424 5681 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7726590f3744e87a84fe02c73e404e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7726590f3744e87a84fe02c73e404e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45049 } }
I20260927 12:41:07.139596 5681 raft_consensus.cc:388] T 00000000000000000000000000000000 P d7726590f3744e87a84fe02c73e404e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:41:07.139683 5681 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7726590f3744e87a84fe02c73e404e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:41:07.139798 5681 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7726590f3744e87a84fe02c73e404e1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:41:07.140708 5681 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7726590f3744e87a84fe02c73e404e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7726590f3744e87a84fe02c73e404e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45049 } }
I20260927 12:41:07.141040 5681 leader_election.cc:302] T 00000000000000000000000000000000 P d7726590f3744e87a84fe02c73e404e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7726590f3744e87a84fe02c73e404e1; no voters:
I20260927 12:41:07.141340 5681 leader_election.cc:288] T 00000000000000000000000000000000 P d7726590f3744e87a84fe02c73e404e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:41:07.141458 5686 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7726590f3744e87a84fe02c73e404e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:41:07.141741 5686 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7726590f3744e87a84fe02c73e404e1 [term 1 LEADER]: Becoming Leader. State: Replica: d7726590f3744e87a84fe02c73e404e1, State: Running, Role: LEADER
I20260927 12:41:07.142107 5686 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7726590f3744e87a84fe02c73e404e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7726590f3744e87a84fe02c73e404e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45049 } }
I20260927 12:41:07.142616 5681 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7726590f3744e87a84fe02c73e404e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:41:07.143697 5688 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7726590f3744e87a84fe02c73e404e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7726590f3744e87a84fe02c73e404e1. Latest consensus state: current_term: 1 leader_uuid: "d7726590f3744e87a84fe02c73e404e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7726590f3744e87a84fe02c73e404e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45049 } } }
I20260927 12:41:07.143832 5688 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7726590f3744e87a84fe02c73e404e1 [sys.catalog]: This master's current role is: LEADER
I20260927 12:41:07.144219 5695 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:41:07.144171 5687 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7726590f3744e87a84fe02c73e404e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d7726590f3744e87a84fe02c73e404e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7726590f3744e87a84fe02c73e404e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45049 } } }
I20260927 12:41:07.144348 5687 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7726590f3744e87a84fe02c73e404e1 [sys.catalog]: This master's current role is: LEADER
I20260927 12:41:07.147521 5695 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:41:07.152948 5695 catalog_manager.cc:1347] Generated new cluster ID: 8cd83c835b934731972a53a30a64c00c
I20260927 12:41:07.153012 5695 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:41:07.167514 5695 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:41:07.168385 5695 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:41:07.175606 5695 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d7726590f3744e87a84fe02c73e404e1: Generated new TSK 0
I20260927 12:41:07.176409 5695 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 12:41:07.211716 5711 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260927 12:41:07.211822 5711 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 12:41:07.212590 5684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:07.190153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53438 (local address 127.0.98.62:45049)
0927 12:41:07.190278 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:07.190282 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:07.190410 (+ 128us) server_negotiation.cc:407] Connection header received
0927 12:41:07.190561 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:07.190563 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:07.190605 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:07.190689 (+ 84us) server_negotiation.cc:226] Negotiated authn=JWT
0927 12:41:07.190937 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:07.191446 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:07.192674 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:07.192892 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:07.192962 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:07.193306 (+ 344us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 12:41:07.212144 (+ 18838us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40239/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 12:41:07.212441 (+ 297us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53438: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40239/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":45}
W20260927 12:41:07.212669 5684 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53438: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40239/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260927 12:41:07.212702 5710 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:41:07.189986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45049 (local address 127.0.0.1:53438)
0927 12:41:07.190306 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:07.190320 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:07.190449 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:07.190710 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:07.190713 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:07.190721 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0927 12:41:07.190839 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:07.190845 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:07.191545 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:07.191549 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:07.192556 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:07.192566 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:07.192998 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:07.193018 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:07.193118 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:07.193146 (+ 28us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 12:41:07.212367 (+ 19221us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40239/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 12:41:07.212521 (+ 154us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:45049: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40239/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":111,"threads_started":1}
I20260927 12:41:07.213326 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 5614
2026-09-27T12:41:07Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (313 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260927 12:41:07.252348 392 mini_oidc.cc:152] Starting JWKS server
I20260927 12:41:07.253165 392 webserver.cc:460] Webserver started at https://127.0.0.1:39303/ using document root <none> and password file <none>
I20260927 12:41:07.253243 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 12:41:07.253504 392 webserver.cc:460] Webserver started at http://127.0.0.1:32961/ using document root <none> and password file <none>
2026-09-27T12:41:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:41:07Z Disabled control of system clock
I20260927 12:41:07.262329 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790512840836255-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790512840836255-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790512840836255-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44539
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44539
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39303/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 12:41:07.369285 5717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:41:07.369556 5717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:41:07.369675 5717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:41:07.373621 5717 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:41:07.373674 5717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:41:07.373690 5717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:41:07.373706 5717 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:41:07.373721 5717 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 12:41:07.373737 5717 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 12:41:07.373750 5717 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39303/jwks.json
I20260927 12:41:07.378460 5717 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790512840836255-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790512840836255-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44539
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39303/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5717
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790512840836255-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:41:07.379452 5717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:41:07.380746 5717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:41:07.389313 5723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:07.389356 5725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:07.389441 5717 server_base.cc:1034] running on GCE node
W20260927 12:41:07.389313 5722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:07.390136 5717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:41:07.391193 5717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:41:07.392346 5717 hybrid_clock.cc:648] HybridClock initialized: now 1790512867392316 us; error 50 us; skew 500 ppm
I20260927 12:41:07.401069 5717 webserver.cc:460] Webserver started at http://127.0.98.62:35925/ using document root <none> and password file <none>
I20260927 12:41:07.401715 5717 fs_manager.cc:359] Metadata directory not provided
I20260927 12:41:07.401790 5717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:41:07.401965 5717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:41:07.404175 5717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790512840836255-392-0/minicluster-data/master-0/data/instance:
uuid: "c6f51bf824424518b9e8406b9e66e26f"
format_stamp: "Formatted at 2026-09-27 12:41:07 on dist-test-slave-bnc1"
server_key: "da45e68a215fc6f3475012215e05e204"
server_key_iv: "4396e23600d9d5fec86d9f6c96288ff0"
server_key_version: "encryptionkey@0"
I20260927 12:41:07.404723 5717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790512840836255-392-0/minicluster-data/master-0/wal/instance:
uuid: "c6f51bf824424518b9e8406b9e66e26f"
format_stamp: "Formatted at 2026-09-27 12:41:07 on dist-test-slave-bnc1"
server_key: "da45e68a215fc6f3475012215e05e204"
server_key_iv: "4396e23600d9d5fec86d9f6c96288ff0"
server_key_version: "encryptionkey@0"
I20260927 12:41:07.408458 5717 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 12:41:07.410877 5731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:07.412122 5717 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 12:41:07.412220 5717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790512840836255-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790512840836255-392-0/minicluster-data/master-0/wal
uuid: "c6f51bf824424518b9e8406b9e66e26f"
format_stamp: "Formatted at 2026-09-27 12:41:07 on dist-test-slave-bnc1"
server_key: "da45e68a215fc6f3475012215e05e204"
server_key_iv: "4396e23600d9d5fec86d9f6c96288ff0"
server_key_version: "encryptionkey@0"
I20260927 12:41:07.412295 5717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790512840836255-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790512840836255-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790512840836255-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:41:07.431483 5717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:41:07.432077 5717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:41:07.432219 5717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:41:07.438820 5717 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44539
I20260927 12:41:07.438886 5783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44539 every 8 connection(s)
I20260927 12:41:07.439749 5717 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:41:07.442900 5784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:41:07.447535 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 5717
I20260927 12:41:07.447670 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790512840836255-392-0/minicluster-data/master-0/wal/instance
I20260927 12:41:07.447945 392 external_mini_cluster.cc:1428] Setting key f06fcca00b75ecd96d7a380b742fc82e
I20260927 12:41:07.448654 5784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6f51bf824424518b9e8406b9e66e26f: Bootstrap starting.
I20260927 12:41:07.451114 5784 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6f51bf824424518b9e8406b9e66e26f: Neither blocks nor log segments found. Creating new log.
I20260927 12:41:07.451828 5784 log.cc:824] T 00000000000000000000000000000000 P c6f51bf824424518b9e8406b9e66e26f: Log is configured to *not* fsync() on all Append() calls
I20260927 12:41:07.453795 5784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6f51bf824424518b9e8406b9e66e26f: No bootstrap required, opened a new log
I20260927 12:41:07.457368 5787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:07.449529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47290 (local address 127.0.98.62:44539)
0927 12:41:07.450265 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:07.450278 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:07.450321 (+ 43us) server_negotiation.cc:407] Connection header received
0927 12:41:07.450936 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:07.450954 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:07.451418 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:07.451770 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:07.452187 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:07.453202 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:07.454196 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:07.454680 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:07.454850 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:07.455705 (+ 855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:07.455709 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:07.455722 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:41:07.456133 (+ 411us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:07.456185 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:41:07.456204 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:07.456311 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:07.456450 (+ 139us) server_negotiation.cc:299] Negotiation successful
0927 12:41:07.456687 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":137,"threads_started":1}
I20260927 12:41:07.457392 5784 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6f51bf824424518b9e8406b9e66e26f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6f51bf824424518b9e8406b9e66e26f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44539 } }
I20260927 12:41:07.457741 5784 raft_consensus.cc:374] T 00000000000000000000000000000000 P c6f51bf824424518b9e8406b9e66e26f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:41:07.457818 5784 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6f51bf824424518b9e8406b9e66e26f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6f51bf824424518b9e8406b9e66e26f, State: Initialized, Role: FOLLOWER
I20260927 12:41:07.458230 5784 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6f51bf824424518b9e8406b9e66e26f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6f51bf824424518b9e8406b9e66e26f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44539 } }
I20260927 12:41:07.458374 5784 raft_consensus.cc:388] T 00000000000000000000000000000000 P c6f51bf824424518b9e8406b9e66e26f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:41:07.458441 5784 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6f51bf824424518b9e8406b9e66e26f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:41:07.458550 5784 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6f51bf824424518b9e8406b9e66e26f [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:41:07.459507 5784 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6f51bf824424518b9e8406b9e66e26f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6f51bf824424518b9e8406b9e66e26f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44539 } }
I20260927 12:41:07.459880 5784 leader_election.cc:302] T 00000000000000000000000000000000 P c6f51bf824424518b9e8406b9e66e26f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6f51bf824424518b9e8406b9e66e26f; no voters:
I20260927 12:41:07.460211 5784 leader_election.cc:288] T 00000000000000000000000000000000 P c6f51bf824424518b9e8406b9e66e26f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:41:07.460250 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 12:41:07.460302 5789 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6f51bf824424518b9e8406b9e66e26f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:41:07.460487 5789 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6f51bf824424518b9e8406b9e66e26f [term 1 LEADER]: Becoming Leader. State: Replica: c6f51bf824424518b9e8406b9e66e26f, State: Running, Role: LEADER
I20260927 12:41:07.460788 5789 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6f51bf824424518b9e8406b9e66e26f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6f51bf824424518b9e8406b9e66e26f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44539 } }
I20260927 12:41:07.461472 5784 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6f51bf824424518b9e8406b9e66e26f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:41:07.462078 5791 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6f51bf824424518b9e8406b9e66e26f [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6f51bf824424518b9e8406b9e66e26f. Latest consensus state: current_term: 1 leader_uuid: "c6f51bf824424518b9e8406b9e66e26f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6f51bf824424518b9e8406b9e66e26f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44539 } } }
I20260927 12:41:07.462203 5791 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6f51bf824424518b9e8406b9e66e26f [sys.catalog]: This master's current role is: LEADER
I20260927 12:41:07.462560 5795 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:41:07.462852 5790 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6f51bf824424518b9e8406b9e66e26f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c6f51bf824424518b9e8406b9e66e26f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6f51bf824424518b9e8406b9e66e26f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44539 } } }
I20260927 12:41:07.462977 5790 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6f51bf824424518b9e8406b9e66e26f [sys.catalog]: This master's current role is: LEADER
I20260927 12:41:07.465317 5795 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:41:07.470534 5795 catalog_manager.cc:1347] Generated new cluster ID: bb16b755ea1641019c7dd7edc152541f
I20260927 12:41:07.470592 5795 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:41:07.483690 5795 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:41:07.484886 5795 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:41:07.490449 5795 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c6f51bf824424518b9e8406b9e66e26f: Generated new TSK 0
I20260927 12:41:07.491168 5795 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:41:07.505522 5787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:07.500689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47302 (local address 127.0.98.62:44539)
0927 12:41:07.500819 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:07.500824 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:07.501037 (+ 213us) server_negotiation.cc:407] Connection header received
0927 12:41:07.501185 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:07.501188 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:07.501232 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:07.501317 (+ 85us) server_negotiation.cc:226] Negotiated authn=JWT
0927 12:41:07.501560 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:07.502114 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:07.503334 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:07.503541 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:07.503612 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:07.503910 (+ 298us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 12:41:07.505174 (+ 1264us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39303/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 12:41:07.505392 (+ 218us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47302: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39303/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":51}
W20260927 12:41:07.505596 5787 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47302: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39303/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260927 12:41:07.507493 5813 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:41:07.500630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44539 (local address 127.0.0.1:47302)
0927 12:41:07.500944 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:07.500959 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:07.501072 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:07.501341 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:07.501345 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:07.501353 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0927 12:41:07.501460 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:07.501467 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:07.502225 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:07.502229 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:07.503216 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:07.503225 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:07.503633 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:07.503652 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:07.503742 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:07.503770 (+ 28us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 12:41:07.505346 (+ 1576us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39303/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 12:41:07.507327 (+ 1981us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:44539: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39303/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":233,"mutex_wait_us":1731,"thread_start_us":118,"threads_started":1}
I20260927 12:41:07.507916 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 5717
E20260927 12:41:07.516569 5814 webserver.cc:533] Webserver: sslize failed:
E20260927 12:41:07.516651 5814 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-27T12:41:07Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (293 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-27T12:41:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:41:07Z Disabled control of system clock
I20260927 12:41:07.554263 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790512840836255-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790512840836255-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790512840836255-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34341
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:44947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34341
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790512840836255-392-0/testchainca.cert with env {}
W20260927 12:41:07.662006 5818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:41:07.662237 5818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:41:07.662274 5818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:41:07.666177 5818 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:41:07.666224 5818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:41:07.666239 5818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:41:07.666254 5818 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:41:07.666268 5818 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 12:41:07.666283 5818 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 12:41:07.666297 5818 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260927 12:41:07.671386 5818 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790512840836255-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790512840836255-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34341
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790512840836255-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790512840836255-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:41:07.672278 5818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:41:07.673182 5818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:41:07.681407 5823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:07.681450 5824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:07.681527 5818 server_base.cc:1034] running on GCE node
W20260927 12:41:07.681414 5826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:07.682389 5818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:41:07.683462 5818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:41:07.684659 5818 hybrid_clock.cc:648] HybridClock initialized: now 1790512867684630 us; error 46 us; skew 500 ppm
I20260927 12:41:07.686496 5818 webserver.cc:460] Webserver started at http://127.0.98.62:40411/ using document root <none> and password file <none>
I20260927 12:41:07.687083 5818 fs_manager.cc:359] Metadata directory not provided
I20260927 12:41:07.687176 5818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:41:07.687415 5818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:41:07.689360 5818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790512840836255-392-0/minicluster-data/master-0/data/instance:
uuid: "9cbcce264af04a7ca56f9064c7e67596"
format_stamp: "Formatted at 2026-09-27 12:41:07 on dist-test-slave-bnc1"
server_key: "281f58de698fddbfd19d0ba69bac60c6"
server_key_iv: "bb4eb241e91e1693367b0d25cd9eaf7d"
server_key_version: "encryptionkey@0"
I20260927 12:41:07.689966 5818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790512840836255-392-0/minicluster-data/master-0/wal/instance:
uuid: "9cbcce264af04a7ca56f9064c7e67596"
format_stamp: "Formatted at 2026-09-27 12:41:07 on dist-test-slave-bnc1"
server_key: "281f58de698fddbfd19d0ba69bac60c6"
server_key_iv: "bb4eb241e91e1693367b0d25cd9eaf7d"
server_key_version: "encryptionkey@0"
I20260927 12:41:07.693475 5818 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 12:41:07.695956 5832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:07.697139 5818 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:41:07.697324 5818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790512840836255-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790512840836255-392-0/minicluster-data/master-0/wal
uuid: "9cbcce264af04a7ca56f9064c7e67596"
format_stamp: "Formatted at 2026-09-27 12:41:07 on dist-test-slave-bnc1"
server_key: "281f58de698fddbfd19d0ba69bac60c6"
server_key_iv: "bb4eb241e91e1693367b0d25cd9eaf7d"
server_key_version: "encryptionkey@0"
I20260927 12:41:07.697435 5818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790512840836255-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790512840836255-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790512840836255-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:41:07.717988 5818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:41:07.718662 5818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:41:07.718859 5818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:41:07.725493 5818 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34341
I20260927 12:41:07.725515 5884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34341 every 8 connection(s)
I20260927 12:41:07.726449 5818 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:41:07.729377 5885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:41:07.729936 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 5818
I20260927 12:41:07.730110 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790512840836255-392-0/minicluster-data/master-0/wal/instance
I20260927 12:41:07.730412 392 external_mini_cluster.cc:1428] Setting key 023572f443a5f795fbb7218cb1864aec
I20260927 12:41:07.735169 5885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cbcce264af04a7ca56f9064c7e67596: Bootstrap starting.
I20260927 12:41:07.737339 5885 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9cbcce264af04a7ca56f9064c7e67596: Neither blocks nor log segments found. Creating new log.
I20260927 12:41:07.738080 5885 log.cc:824] T 00000000000000000000000000000000 P 9cbcce264af04a7ca56f9064c7e67596: Log is configured to *not* fsync() on all Append() calls
I20260927 12:41:07.739392 5888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:07.731731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48504 (local address 127.0.98.62:34341)
0927 12:41:07.732386 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:07.732401 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:07.732433 (+ 32us) server_negotiation.cc:407] Connection header received
0927 12:41:07.733039 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:07.733055 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:07.733439 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:07.733823 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:07.734274 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:07.735226 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:07.736110 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:07.736589 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:07.736766 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:07.737652 (+ 886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:07.737658 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:07.737690 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:41:07.738051 (+ 361us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:07.738102 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:41:07.738121 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:07.738242 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:07.738367 (+ 125us) server_negotiation.cc:299] Negotiation successful
0927 12:41:07.738728 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":138,"threads_started":1}
I20260927 12:41:07.739874 5885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cbcce264af04a7ca56f9064c7e67596: No bootstrap required, opened a new log
I20260927 12:41:07.742002 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 12:41:07.743512 5885 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9cbcce264af04a7ca56f9064c7e67596 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cbcce264af04a7ca56f9064c7e67596" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34341 } }
I20260927 12:41:07.743728 5885 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9cbcce264af04a7ca56f9064c7e67596 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:41:07.743808 5885 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9cbcce264af04a7ca56f9064c7e67596 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cbcce264af04a7ca56f9064c7e67596, State: Initialized, Role: FOLLOWER
I20260927 12:41:07.744191 5885 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9cbcce264af04a7ca56f9064c7e67596 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cbcce264af04a7ca56f9064c7e67596" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34341 } }
I20260927 12:41:07.744335 5885 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9cbcce264af04a7ca56f9064c7e67596 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:41:07.744416 5885 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9cbcce264af04a7ca56f9064c7e67596 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:41:07.744527 5885 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9cbcce264af04a7ca56f9064c7e67596 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:41:07.745318 5885 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9cbcce264af04a7ca56f9064c7e67596 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cbcce264af04a7ca56f9064c7e67596" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34341 } }
I20260927 12:41:07.745687 5885 leader_election.cc:302] T 00000000000000000000000000000000 P 9cbcce264af04a7ca56f9064c7e67596 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9cbcce264af04a7ca56f9064c7e67596; no voters:
I20260927 12:41:07.745970 5885 leader_election.cc:288] T 00000000000000000000000000000000 P 9cbcce264af04a7ca56f9064c7e67596 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:41:07.746052 5890 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9cbcce264af04a7ca56f9064c7e67596 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:41:07.746346 5890 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9cbcce264af04a7ca56f9064c7e67596 [term 1 LEADER]: Becoming Leader. State: Replica: 9cbcce264af04a7ca56f9064c7e67596, State: Running, Role: LEADER
I20260927 12:41:07.746635 5890 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9cbcce264af04a7ca56f9064c7e67596 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cbcce264af04a7ca56f9064c7e67596" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34341 } }
I20260927 12:41:07.747222 5885 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9cbcce264af04a7ca56f9064c7e67596 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:41:07.748256 5892 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cbcce264af04a7ca56f9064c7e67596 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9cbcce264af04a7ca56f9064c7e67596. Latest consensus state: current_term: 1 leader_uuid: "9cbcce264af04a7ca56f9064c7e67596" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cbcce264af04a7ca56f9064c7e67596" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34341 } } }
I20260927 12:41:07.748381 5892 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cbcce264af04a7ca56f9064c7e67596 [sys.catalog]: This master's current role is: LEADER
I20260927 12:41:07.748747 5899 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:41:07.748697 5891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cbcce264af04a7ca56f9064c7e67596 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9cbcce264af04a7ca56f9064c7e67596" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cbcce264af04a7ca56f9064c7e67596" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34341 } } }
I20260927 12:41:07.749347 5891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cbcce264af04a7ca56f9064c7e67596 [sys.catalog]: This master's current role is: LEADER
I20260927 12:41:07.751874 5899 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:41:07.757301 5899 catalog_manager.cc:1347] Generated new cluster ID: d0a9f20f0cfb4ee9ac2d0c9767c69427
I20260927 12:41:07.757369 5899 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:41:07.789230 5899 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:41:07.790529 5899 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:41:07.799921 5899 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9cbcce264af04a7ca56f9064c7e67596: Generated new TSK 0
I20260927 12:41:07.800633 5899 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:41:08.285910 5888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:07.812485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48514 (local address 127.0.98.62:34341)
0927 12:41:07.812600 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:07.812603 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:07.812833 (+ 230us) server_negotiation.cc:407] Connection header received
0927 12:41:07.813023 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:07.813025 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:07.813070 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:07.813179 (+ 109us) server_negotiation.cc:226] Negotiated authn=JWT
0927 12:41:07.813415 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:07.814011 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:07.814863 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:07.815101 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:07.815211 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:07.815506 (+ 295us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 12:41:08.285364 (+469858us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 12:41:08.285752 (+ 388us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48514: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":42}
W20260927 12:41:08.286005 5888 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48514: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260927 12:41:08.286021 5914 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:41:07.812417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34341 (local address 127.0.0.1:48514)
0927 12:41:07.812734 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:07.812746 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:07.812887 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:07.813179 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:07.813182 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:07.813190 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0927 12:41:07.813309 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:07.813314 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:07.814134 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:07.814140 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:07.814741 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:07.814747 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:07.815201 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:07.815218 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:07.815315 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:07.815336 (+ 21us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 12:41:08.285648 (+470312us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 12:41:08.285833 (+ 185us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:34341: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":109,"threads_started":1}
I20260927 12:41:08.286661 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 5818
2026-09-27T12:41:08Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (759 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (126 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (119 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:41:08 dist-test-slave-bnc1 krb5kdc[5926](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:41:08 dist-test-slave-bnc1 krb5kdc[5926](info): set up 2 sockets
Sep 27 12:41:08 dist-test-slave-bnc1 krb5kdc[5926](info): commencing operation
krb5kdc: starting...
W20260927 12:41:09.589993 392 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/krb5kdc/test-user.keytab.
Sep 27 12:41:09 dist-test-slave-bnc1 krb5kdc[5926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:41:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:41:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:41:09.712713 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35583
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35583
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:41:09.822235 5939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:41:09.822517 5939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:41:09.822588 5939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:41:09.826584 5939 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:41:09.826673 5939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:41:09.826721 5939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:41:09.826777 5939 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:41:09.826814 5939 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:41:09.832032 5939 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35583
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:41:09.833062 5939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:41:09.834096 5939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:41:09.842900 5944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:09.842927 5947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:09.843081 5939 server_base.cc:1034] running on GCE node
W20260927 12:41:09.843086 5945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:09.843768 5939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:41:09.844949 5939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:41:09.846144 5939 hybrid_clock.cc:648] HybridClock initialized: now 1790512869846122 us; error 31 us; skew 500 ppm
Sep 27 12:41:09 dist-test-slave-bnc1 krb5kdc[5926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512869, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:41:09.849220 5939 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:41:09.850230 5939 webserver.cc:460] Webserver started at http://127.0.98.62:33441/ using document root <none> and password file <none>
I20260927 12:41:09.850798 5939 fs_manager.cc:359] Metadata directory not provided
I20260927 12:41:09.850888 5939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:41:09.851125 5939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:41:09.853175 5939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/minicluster-data/master-0/data/instance:
uuid: "918151a980794d68b3fdea0633e61e9c"
format_stamp: "Formatted at 2026-09-27 12:41:09 on dist-test-slave-bnc1"
server_key: "6a3eacc41f3f16b4b43e34484c7414d7"
server_key_iv: "6ced4dd90ccc2f9681f994659d26da8b"
server_key_version: "encryptionkey@0"
I20260927 12:41:09.853816 5939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/minicluster-data/master-0/wal/instance:
uuid: "918151a980794d68b3fdea0633e61e9c"
format_stamp: "Formatted at 2026-09-27 12:41:09 on dist-test-slave-bnc1"
server_key: "6a3eacc41f3f16b4b43e34484c7414d7"
server_key_iv: "6ced4dd90ccc2f9681f994659d26da8b"
server_key_version: "encryptionkey@0"
I20260927 12:41:09.857445 5939 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 12:41:09.859808 5954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:09.861016 5939 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:41:09.861161 5939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/minicluster-data/master-0/wal
uuid: "918151a980794d68b3fdea0633e61e9c"
format_stamp: "Formatted at 2026-09-27 12:41:09 on dist-test-slave-bnc1"
server_key: "6a3eacc41f3f16b4b43e34484c7414d7"
server_key_iv: "6ced4dd90ccc2f9681f994659d26da8b"
server_key_version: "encryptionkey@0"
I20260927 12:41:09.861270 5939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:41:09.887936 5939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:41:09.891307 5939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:41:09.891520 5939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:41:09.898671 5939 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35583
I20260927 12:41:09.898955 6006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35583 every 8 connection(s)
I20260927 12:41:09.899720 5939 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:41:09.902447 6007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:41:09.907853 6007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 918151a980794d68b3fdea0633e61e9c: Bootstrap starting.
I20260927 12:41:09.909333 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 5939
I20260927 12:41:09.909466 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/minicluster-data/master-0/wal/instance
I20260927 12:41:09.909766 392 external_mini_cluster.cc:1428] Setting key 401486ee35153c9e9e141e62665e3efd
I20260927 12:41:09.910632 6007 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 918151a980794d68b3fdea0633e61e9c: Neither blocks nor log segments found. Creating new log.
I20260927 12:41:09.911533 6007 log.cc:824] T 00000000000000000000000000000000 P 918151a980794d68b3fdea0633e61e9c: Log is configured to *not* fsync() on all Append() calls
I20260927 12:41:09.913342 6007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 918151a980794d68b3fdea0633e61e9c: No bootstrap required, opened a new log
I20260927 12:41:09.917755 6007 raft_consensus.cc:348] T 00000000000000000000000000000000 P 918151a980794d68b3fdea0633e61e9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "918151a980794d68b3fdea0633e61e9c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35583 } }
I20260927 12:41:09.918051 6007 raft_consensus.cc:374] T 00000000000000000000000000000000 P 918151a980794d68b3fdea0633e61e9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:41:09.918134 6007 raft_consensus.cc:729] T 00000000000000000000000000000000 P 918151a980794d68b3fdea0633e61e9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 918151a980794d68b3fdea0633e61e9c, State: Initialized, Role: FOLLOWER
I20260927 12:41:09.918694 6007 consensus_queue.cc:260] T 00000000000000000000000000000000 P 918151a980794d68b3fdea0633e61e9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "918151a980794d68b3fdea0633e61e9c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35583 } }
I20260927 12:41:09.918885 6007 raft_consensus.cc:388] T 00000000000000000000000000000000 P 918151a980794d68b3fdea0633e61e9c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:41:09.918967 6007 raft_consensus.cc:482] T 00000000000000000000000000000000 P 918151a980794d68b3fdea0633e61e9c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:41:09.919062 6007 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 918151a980794d68b3fdea0633e61e9c [term 0 FOLLOWER]: Advancing to term 1
Sep 27 12:41:09 dist-test-slave-bnc1 krb5kdc[5926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:41:09.920260 6007 raft_consensus.cc:504] T 00000000000000000000000000000000 P 918151a980794d68b3fdea0633e61e9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "918151a980794d68b3fdea0633e61e9c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35583 } }
I20260927 12:41:09.920650 6007 leader_election.cc:302] T 00000000000000000000000000000000 P 918151a980794d68b3fdea0633e61e9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 918151a980794d68b3fdea0633e61e9c; no voters:
I20260927 12:41:09.921033 6007 leader_election.cc:288] T 00000000000000000000000000000000 P 918151a980794d68b3fdea0633e61e9c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:41:09.921147 6012 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 918151a980794d68b3fdea0633e61e9c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:41:09.921389 6012 raft_consensus.cc:686] T 00000000000000000000000000000000 P 918151a980794d68b3fdea0633e61e9c [term 1 LEADER]: Becoming Leader. State: Replica: 918151a980794d68b3fdea0633e61e9c, State: Running, Role: LEADER
I20260927 12:41:09.921780 6012 consensus_queue.cc:237] T 00000000000000000000000000000000 P 918151a980794d68b3fdea0633e61e9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "918151a980794d68b3fdea0633e61e9c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35583 } }
I20260927 12:41:09.922473 6007 sys_catalog.cc:564] T 00000000000000000000000000000000 P 918151a980794d68b3fdea0633e61e9c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:41:09.923532 6014 sys_catalog.cc:455] T 00000000000000000000000000000000 P 918151a980794d68b3fdea0633e61e9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 918151a980794d68b3fdea0633e61e9c. Latest consensus state: current_term: 1 leader_uuid: "918151a980794d68b3fdea0633e61e9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "918151a980794d68b3fdea0633e61e9c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35583 } } }
I20260927 12:41:09.923676 6014 sys_catalog.cc:458] T 00000000000000000000000000000000 P 918151a980794d68b3fdea0633e61e9c [sys.catalog]: This master's current role is: LEADER
I20260927 12:41:09.924034 6013 sys_catalog.cc:455] T 00000000000000000000000000000000 P 918151a980794d68b3fdea0633e61e9c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "918151a980794d68b3fdea0633e61e9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "918151a980794d68b3fdea0633e61e9c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35583 } } }
I20260927 12:41:09.924113 6021 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:41:09.924130 6013 sys_catalog.cc:458] T 00000000000000000000000000000000 P 918151a980794d68b3fdea0633e61e9c [sys.catalog]: This master's current role is: LEADER
I20260927 12:41:09.926205 6010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:09.910918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34812 (local address 127.0.98.62:35583)
0927 12:41:09.911351 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:09.911365 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:09.911398 (+ 33us) server_negotiation.cc:407] Connection header received
0927 12:41:09.912195 (+ 797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:09.912219 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:09.912543 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:09.912877 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:09.913697 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:09.915186 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:09.915994 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:09.916796 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:09.916981 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:09.920077 (+ 3096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:09.920082 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:09.920096 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:41:09.920121 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:09.922223 (+ 2102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:09.922767 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:09.922773 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:09.922779 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:09.922932 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:09.923273 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:09.923276 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:09.923278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:09.923869 (+ 591us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:09.924791 (+ 922us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:09.924946 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 12:41:09.925242 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":126,"threads_started":1}
I20260927 12:41:09.927280 6021 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:41:09.933153 6021 catalog_manager.cc:1347] Generated new cluster ID: 9819757e5ce049abba6ac20b2aed0ba0
I20260927 12:41:09.933239 6021 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:41:09.941406 6021 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:41:09.942303 6021 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:41:09.953747 6021 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 918151a980794d68b3fdea0633e61e9c: Generated new TSK 0
I20260927 12:41:09.954563 6021 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 12:41:09.986825 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35583
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:41:10.097265 6033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:41:10.097564 6033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:41:10.097669 6033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:41:10.101715 6033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:41:10.101812 6033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:41:10.101913 6033 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:41:10.107273 6033 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:41:10.108349 6033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:41:10.109673 6033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:41:10.118839 6039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:10.118904 6041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:10.118880 6038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:10.119225 6033 server_base.cc:1034] running on GCE node
I20260927 12:41:10.119571 6033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:41:10.120146 6033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:41:10.121313 6033 hybrid_clock.cc:648] HybridClock initialized: now 1790512870121299 us; error 26 us; skew 500 ppm
Sep 27 12:41:10 dist-test-slave-bnc1 krb5kdc[5926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512870, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:41:10.124193 6033 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 12:41:10.125023 6033 webserver.cc:460] Webserver started at http://127.0.98.1:37943/ using document root <none> and password file <none>
I20260927 12:41:10.125644 6033 fs_manager.cc:359] Metadata directory not provided
I20260927 12:41:10.125707 6033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:41:10.125924 6033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:41:10.127915 6033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/minicluster-data/ts-0/data/instance:
uuid: "d63454d636764d93b541130f1768a1eb"
format_stamp: "Formatted at 2026-09-27 12:41:10 on dist-test-slave-bnc1"
server_key: "ee6626ba19339906d975cc22e5b1b894"
server_key_iv: "39031f3c311c5853729411eb5ea6ac8b"
server_key_version: "encryptionkey@0"
I20260927 12:41:10.128484 6033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d63454d636764d93b541130f1768a1eb"
format_stamp: "Formatted at 2026-09-27 12:41:10 on dist-test-slave-bnc1"
server_key: "ee6626ba19339906d975cc22e5b1b894"
server_key_iv: "39031f3c311c5853729411eb5ea6ac8b"
server_key_version: "encryptionkey@0"
I20260927 12:41:10.132119 6033 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 12:41:10.134512 6048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:10.135723 6033 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 12:41:10.135842 6033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/minicluster-data/ts-0/wal
uuid: "d63454d636764d93b541130f1768a1eb"
format_stamp: "Formatted at 2026-09-27 12:41:10 on dist-test-slave-bnc1"
server_key: "ee6626ba19339906d975cc22e5b1b894"
server_key_iv: "39031f3c311c5853729411eb5ea6ac8b"
server_key_version: "encryptionkey@0"
I20260927 12:41:10.135941 6033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:41:10.158667 6033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:41:10.161391 6033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:41:10.161612 6033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:41:10.162194 6033 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:41:10.163043 6033 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:41:10.163110 6033 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:10.163180 6033 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:41:10.163223 6033 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:10.171938 6033 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39871
I20260927 12:41:10.171991 6161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39871 every 8 connection(s)
I20260927 12:41:10.172834 6033 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:41:10.173635 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 6033
I20260927 12:41:10.173803 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/minicluster-data/ts-0/wal/instance
I20260927 12:41:10.174079 392 external_mini_cluster.cc:1428] Setting key c44c0c903319b32cf35fe608cf9b92be
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Sep 27 12:41:10 dist-test-slave-bnc1 krb5kdc[5926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512870, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 12:41:10.186945 6010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:10.174869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48589 (local address 127.0.98.62:35583)
0927 12:41:10.174996 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:10.175007 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:10.175918 (+ 911us) server_negotiation.cc:407] Connection header received
0927 12:41:10.177280 (+ 1362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:10.177284 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:10.177335 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:10.177415 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:10.178598 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:10.179122 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:10.180256 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:10.180494 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:10.180602 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:10.183056 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:10.183059 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:10.183061 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:41:10.183083 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:10.184506 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:10.185043 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:10.185050 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:10.185055 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:10.185147 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:10.185617 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:10.185623 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:10.185628 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:10.185821 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:10.185914 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:10.186573 (+ 659us) server_negotiation.cc:299] Negotiation successful
0927 12:41:10.186727 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 12:41:10.187698 6164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:10.175297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35583 (local address 127.0.98.1:48589)
0927 12:41:10.175788 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:10.175828 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:10.177059 (+ 1231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:10.177616 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:10.177631 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:10.177983 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:10.178453 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:10.178466 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:10.179255 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:10.179260 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:10.180118 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:10.180126 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:10.180637 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:10.180641 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:10.180810 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:10.181383 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:10.181399 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:10.182903 (+ 1504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:41:10.184607 (+ 1704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:10.184612 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:10.184623 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:10.184928 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:10.185255 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:10.185258 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:10.185260 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:10.185451 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:10.185939 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:41:10.185948 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:41:10.186282 (+ 334us) client_negotiation.cc:769] Sending connection context
0927 12:41:10.186510 (+ 228us) client_negotiation.cc:240] Negotiation successful
0927 12:41:10.186831 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":136,"threads_started":1}
I20260927 12:41:10.189095 6162 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35583
I20260927 12:41:10.189358 6162 heartbeater.cc:464] Registering TS with master...
I20260927 12:41:10.190009 6162 heartbeater.cc:511] Master 127.0.98.62:35583 requested a full tablet report, sending...
I20260927 12:41:10.191730 5971 ts_manager.cc:194] Registered new tserver with Master: d63454d636764d93b541130f1768a1eb (127.0.98.1:39871)
I20260927 12:41:10.192868 5971 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48589
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 12:41:10.201321 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35583
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:41:10.306913 6167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:41:10.307137 6167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:41:10.307176 6167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:41:10.311007 6167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:41:10.311055 6167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:41:10.311108 6167 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:41:10.315917 6167 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6167
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:41:10.316859 6167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:41:10.317797 6167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:41:10.327355 6175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:10.327382 6173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:10.327625 6167 server_base.cc:1034] running on GCE node
W20260927 12:41:10.327339 6172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:10.328195 6167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:41:10.328775 6167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:41:10.329957 6167 hybrid_clock.cc:648] HybridClock initialized: now 1790512870329928 us; error 42 us; skew 500 ppm
Sep 27 12:41:10 dist-test-slave-bnc1 krb5kdc[5926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512870, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:41:10.332535 6167 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 12:41:10.333375 6167 webserver.cc:460] Webserver started at http://127.0.98.2:33301/ using document root <none> and password file <none>
I20260927 12:41:10.333961 6167 fs_manager.cc:359] Metadata directory not provided
I20260927 12:41:10.334045 6167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:41:10.334260 6167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:41:10.336138 6167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/minicluster-data/ts-1/data/instance:
uuid: "2d576a83e49440e6984ba65c9477162b"
format_stamp: "Formatted at 2026-09-27 12:41:10 on dist-test-slave-bnc1"
server_key: "e8eecc753cbc25b14fb7897f68f25674"
server_key_iv: "d277b5de9e25564570e5c10fd037cef5"
server_key_version: "encryptionkey@0"
I20260927 12:41:10.336736 6167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2d576a83e49440e6984ba65c9477162b"
format_stamp: "Formatted at 2026-09-27 12:41:10 on dist-test-slave-bnc1"
server_key: "e8eecc753cbc25b14fb7897f68f25674"
server_key_iv: "d277b5de9e25564570e5c10fd037cef5"
server_key_version: "encryptionkey@0"
I20260927 12:41:10.340260 6167 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 12:41:10.342551 6182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:10.343689 6167 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 12:41:10.343813 6167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/minicluster-data/ts-1/wal
uuid: "2d576a83e49440e6984ba65c9477162b"
format_stamp: "Formatted at 2026-09-27 12:41:10 on dist-test-slave-bnc1"
server_key: "e8eecc753cbc25b14fb7897f68f25674"
server_key_iv: "d277b5de9e25564570e5c10fd037cef5"
server_key_version: "encryptionkey@0"
I20260927 12:41:10.343932 6167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:41:10.391908 6167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:41:10.395313 6167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:41:10.395516 6167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:41:10.396111 6167 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:41:10.396929 6167 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:41:10.396998 6167 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:10.397063 6167 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:41:10.397113 6167 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:10.406663 6167 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36217
I20260927 12:41:10.406699 6295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36217 every 8 connection(s)
I20260927 12:41:10.407594 6167 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
Sep 27 12:41:10 dist-test-slave-bnc1 krb5kdc[5926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512870, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:41:10.417553 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 6167
I20260927 12:41:10.417732 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/minicluster-data/ts-1/wal/instance
I20260927 12:41:10.418035 392 external_mini_cluster.cc:1428] Setting key c2c4e65f16960f9b659da35542d87c5e
I20260927 12:41:10.421629 6010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:10.409428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43813 (local address 127.0.98.62:35583)
0927 12:41:10.409557 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:10.409560 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:10.410430 (+ 870us) server_negotiation.cc:407] Connection header received
0927 12:41:10.411809 (+ 1379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:10.411813 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:10.411871 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:10.411952 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:10.413188 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:10.413798 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:10.414959 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:10.415186 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:10.415306 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:10.417817 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:10.417824 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:10.417830 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:41:10.417859 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:10.419350 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:10.419920 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:10.419925 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:10.419928 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:10.420023 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:10.420404 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:10.420410 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:10.420414 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:10.420666 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:10.420769 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:10.421310 (+ 541us) server_negotiation.cc:299] Negotiation successful
0927 12:41:10.421437 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 12:41:10.422480 6298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:10.409839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35583 (local address 127.0.98.2:43813)
0927 12:41:10.410273 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:10.410316 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:10.411597 (+ 1281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:10.412113 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:10.412120 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:10.412556 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:10.413038 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:10.413052 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:10.413922 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:10.413928 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:10.414834 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:10.414843 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:10.415332 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:10.415337 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:10.415507 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:10.416060 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:10.416076 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:10.417522 (+ 1446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:41:10.419470 (+ 1948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:10.419476 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:10.419491 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:10.419788 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:10.420136 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:10.420138 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:10.420139 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:10.420307 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:10.420765 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:41:10.420771 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:41:10.420991 (+ 220us) client_negotiation.cc:769] Sending connection context
0927 12:41:10.421268 (+ 277us) client_negotiation.cc:240] Negotiation successful
0927 12:41:10.421618 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":114,"threads_started":1}
I20260927 12:41:10.423817 6296 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35583
I20260927 12:41:10.424096 6296 heartbeater.cc:464] Registering TS with master...
I20260927 12:41:10.424642 6296 heartbeater.cc:511] Master 127.0.98.62:35583 requested a full tablet report, sending...
I20260927 12:41:10.425815 5971 ts_manager.cc:194] Registered new tserver with Master: 2d576a83e49440e6984ba65c9477162b (127.0.98.2:36217)
I20260927 12:41:10.426623 5971 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43813
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 12:41:10.445016 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35583
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:41:10.550556 6301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:41:10.550812 6301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:41:10.550884 6301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:41:10.554908 6301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:41:10.554992 6301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:41:10.555080 6301 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:41:10.560967 6301 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:41:10.562004 6301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:41:10.563014 6301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:41:10.572186 6306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:10.572186 6309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:10.572173 6307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:10.572467 6301 server_base.cc:1034] running on GCE node
I20260927 12:41:10.572932 6301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:41:10.573541 6301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:41:10.574762 6301 hybrid_clock.cc:648] HybridClock initialized: now 1790512870574736 us; error 36 us; skew 500 ppm
Sep 27 12:41:10 dist-test-slave-bnc1 krb5kdc[5926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512870, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:41:10.580951 6301 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 12:41:10.581820 6301 webserver.cc:460] Webserver started at http://127.0.98.3:33373/ using document root <none> and password file <none>
I20260927 12:41:10.582330 6301 fs_manager.cc:359] Metadata directory not provided
I20260927 12:41:10.582394 6301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:41:10.582549 6301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:41:10.584519 6301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/minicluster-data/ts-2/data/instance:
uuid: "d854f6b0256d4630834147dc8f1c268a"
format_stamp: "Formatted at 2026-09-27 12:41:10 on dist-test-slave-bnc1"
server_key: "e53975af344e615852a78e272d3eda64"
server_key_iv: "be920d8e734197a80b0ac0ae7d0b9d02"
server_key_version: "encryptionkey@0"
I20260927 12:41:10.585022 6301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d854f6b0256d4630834147dc8f1c268a"
format_stamp: "Formatted at 2026-09-27 12:41:10 on dist-test-slave-bnc1"
server_key: "e53975af344e615852a78e272d3eda64"
server_key_iv: "be920d8e734197a80b0ac0ae7d0b9d02"
server_key_version: "encryptionkey@0"
I20260927 12:41:10.588698 6301 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 12:41:10.591094 6316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:10.592307 6301 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260927 12:41:10.592411 6301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/minicluster-data/ts-2/wal
uuid: "d854f6b0256d4630834147dc8f1c268a"
format_stamp: "Formatted at 2026-09-27 12:41:10 on dist-test-slave-bnc1"
server_key: "e53975af344e615852a78e272d3eda64"
server_key_iv: "be920d8e734197a80b0ac0ae7d0b9d02"
server_key_version: "encryptionkey@0"
I20260927 12:41:10.592495 6301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:41:10.610231 6301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:41:10.615692 6301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:41:10.615988 6301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:41:10.616783 6301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:41:10.617906 6301 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:41:10.617969 6301 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:10.618069 6301 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:41:10.618124 6301 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:10.629911 6301 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33353
I20260927 12:41:10.629943 6429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33353 every 8 connection(s)
I20260927 12:41:10.631018 6301 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:41:10.640561 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 6301
I20260927 12:41:10.640692 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/minicluster-data/ts-2/wal/instance
I20260927 12:41:10.641029 392 external_mini_cluster.cc:1428] Setting key cf135f851e644b72788da40d0714f04e
Sep 27 12:41:10 dist-test-slave-bnc1 krb5kdc[5926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512870, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:41:10.653522 6010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:10.633761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48773 (local address 127.0.98.62:35583)
0927 12:41:10.633900 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:10.633905 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:10.634546 (+ 641us) server_negotiation.cc:407] Connection header received
0927 12:41:10.636587 (+ 2041us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:10.636590 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:10.636634 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:10.636759 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:10.638475 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:10.639148 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:10.640493 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:10.640730 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:10.640842 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:10.649722 (+ 8880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:10.649726 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:10.649729 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:41:10.649754 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:10.651285 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:10.651905 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:10.651908 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:10.651910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:10.651995 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:10.652333 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:10.652336 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:10.652337 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:10.652573 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:10.652671 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:10.653240 (+ 569us) server_negotiation.cc:299] Negotiation successful
0927 12:41:10.653354 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 12:41:10.654517 6432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:10.633786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35583 (local address 127.0.98.3:48773)
0927 12:41:10.634347 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:10.634402 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:10.636339 (+ 1937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:10.636970 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:10.636982 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:10.637522 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:10.638304 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:10.638326 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:10.639335 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:10.639337 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:10.640359 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:10.640371 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:10.640894 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:10.640899 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:10.641060 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:10.641717 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:10.641737 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:10.643614 (+ 1877us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:41:10.651433 (+ 7819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:10.651438 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:10.651453 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:10.651813 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:10.652094 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:10.652097 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:10.652098 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:10.652234 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:10.652679 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:41:10.652688 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:41:10.652996 (+ 308us) client_negotiation.cc:769] Sending connection context
0927 12:41:10.653185 (+ 189us) client_negotiation.cc:240] Negotiation successful
0927 12:41:10.653493 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":142,"threads_started":1}
I20260927 12:41:10.655864 6430 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35583
I20260927 12:41:10.656097 6430 heartbeater.cc:464] Registering TS with master...
I20260927 12:41:10.656560 6430 heartbeater.cc:511] Master 127.0.98.62:35583 requested a full tablet report, sending...
I20260927 12:41:10.657493 5971 ts_manager.cc:194] Registered new tserver with Master: d854f6b0256d4630834147dc8f1c268a (127.0.98.3:33353)
I20260927 12:41:10.658052 5971 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48773
I20260927 12:41:10.667415 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/krb5kdc/test-admin.keytab.
Sep 27 12:41:10 dist-test-slave-bnc1 krb5kdc[5926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:41:10.682531 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260927 12:41:10.685124 6437 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:41:10.683787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34552 (local address 127.0.98.1:39871)
0927 12:41:10.684057 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:10.684070 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:10.684108 (+ 38us) server_negotiation.cc:407] Connection header received
0927 12:41:10.684153 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:10.684158 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:10.684335 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:10.684446 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:10.684935 (+ 489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34552: BlockingRecv error: recv got EOF from 127.0.0.1:34552 (error 108)
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":79,"threads_started":1}
W20260927 12:41:10.685146 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/krb5kdc/krb5cc' of type FILE
Sep 27 12:41:10 dist-test-slave-bnc1 krb5kdc[5926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:41:10.686349 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 12:41:10 dist-test-slave-bnc1 krb5kdc[5926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:41:10.687441 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 12:41:10 dist-test-slave-bnc1 krb5kdc[5926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 12:41:10.697134 6437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:10.688441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34566 (local address 127.0.98.1:39871)
0927 12:41:10.688554 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:10.688557 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:10.688715 (+ 158us) server_negotiation.cc:407] Connection header received
0927 12:41:10.688842 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:10.688844 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:10.688880 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:10.688940 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:10.689535 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:10.690299 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:10.691217 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:10.691559 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:10.691638 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:10.694046 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:10.694055 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:10.694067 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:41:10.694088 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:10.695490 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:10.695974 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:10.695979 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:10.695980 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:10.696053 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:10.696268 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:10.696270 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:10.696271 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:10.696466 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:10.696606 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:10.696836 (+ 230us) server_negotiation.cc:299] Negotiation successful
0927 12:41:10.697018 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260927 12:41:10.700914 6437 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:41:10.699893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34574 (local address 127.0.98.1:39871)
0927 12:41:10.700009 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:10.700014 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:10.700120 (+ 106us) server_negotiation.cc:407] Connection header received
0927 12:41:10.700261 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:10.700265 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:10.700309 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:10.700394 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:10.700804 (+ 410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34574: BlockingRecv error: recv got EOF from 127.0.0.1:34574 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260927 12:41:10.701073 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/krb5kdc/krb5cc' of type FILE
Sep 27 12:41:10 dist-test-slave-bnc1 krb5kdc[5926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:41:10.702049 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 12:41:10 dist-test-slave-bnc1 krb5kdc[5926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:41:10.702879 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 12:41:10 dist-test-slave-bnc1 krb5kdc[5926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 12:41:10.710425 6437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:10.703641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34588 (local address 127.0.98.1:39871)
0927 12:41:10.703759 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:10.703762 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:10.703804 (+ 42us) server_negotiation.cc:407] Connection header received
0927 12:41:10.703888 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:10.703891 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:10.703922 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:10.703985 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:10.704424 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:10.704904 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:10.705698 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:10.705908 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:10.705983 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:10.707866 (+ 1883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:10.707869 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:10.707871 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:41:10.707890 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:10.709067 (+ 1177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:10.709473 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:10.709476 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:10.709478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:10.709539 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:10.709829 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:10.709831 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:10.709832 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:10.709965 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:10.710056 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:10.710250 (+ 194us) server_negotiation.cc:299] Negotiation successful
0927 12:41:10.710328 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260927 12:41:10.712829 6437 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:41:10.711908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34594 (local address 127.0.98.1:39871)
0927 12:41:10.712017 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:10.712021 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:10.712133 (+ 112us) server_negotiation.cc:407] Connection header received
0927 12:41:10.712247 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:10.712250 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:10.712291 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:10.712368 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:10.712723 (+ 355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34594: BlockingRecv error: recv got EOF from 127.0.0.1:34594 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260927 12:41:10.712961 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790512840836255-392-0/krb5kdc/krb5cc' of type FILE
Sep 27 12:41:10 dist-test-slave-bnc1 krb5kdc[5926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:41:10.713933 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 12:41:10 dist-test-slave-bnc1 krb5kdc[5926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:41:10.714730 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 12:41:10 dist-test-slave-bnc1 krb5kdc[5926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 12:41:10.722453 6437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:10.715373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34606 (local address 127.0.98.1:39871)
0927 12:41:10.715517 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:10.715520 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:10.715585 (+ 65us) server_negotiation.cc:407] Connection header received
0927 12:41:10.715661 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:10.715663 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:10.715690 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:10.715748 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:10.716181 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:10.716652 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:10.717429 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:10.717691 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:10.717766 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:10.719737 (+ 1971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:10.719739 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:10.719741 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:41:10.719760 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:10.721064 (+ 1304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:10.721405 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:10.721407 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:10.721408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:10.721469 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:10.721737 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:10.721739 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:10.721740 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:10.721871 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:10.721972 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:10.722154 (+ 182us) server_negotiation.cc:299] Negotiation successful
0927 12:41:10.722309 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 12:41:10.723328 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 6033
I20260927 12:41:10.729512 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 6167
I20260927 12:41:10.734668 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 6301
I20260927 12:41:10.739765 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 5939
2026-09-27T12:41:10Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2208 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:41:10 dist-test-slave-bnc1 krb5kdc[6450](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:41:10 dist-test-slave-bnc1 krb5kdc[6450](info): set up 2 sockets
Sep 27 12:41:10 dist-test-slave-bnc1 krb5kdc[6450](info): commencing operation
krb5kdc: starting...
W20260927 12:41:11.781988 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/krb5kdc/test-user.keytab.
Sep 27 12:41:11 dist-test-slave-bnc1 krb5kdc[6450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:41:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:41:11Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260927 12:41:11.900564 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46303
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46303
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:41:12.008831 6463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:41:12.009101 6463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:41:12.009145 6463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:41:12.013499 6463 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:41:12.013550 6463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:41:12.013584 6463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:41:12.013602 6463 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:41:12.013616 6463 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:41:12.018779 6463 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46303
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:41:12.019774 6463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:41:12.020759 6463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:41:12.029088 6471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:12.029281 6468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:12.029167 6463 server_base.cc:1034] running on GCE node
W20260927 12:41:12.029120 6469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:12.030081 6463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:41:12.031371 6463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:41:12.032541 6463 hybrid_clock.cc:648] HybridClock initialized: now 1790512872032519 us; error 31 us; skew 500 ppm
Sep 27 12:41:12 dist-test-slave-bnc1 krb5kdc[6450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512872, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:41:12.035264 6463 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260927 12:41:12.036137 6463 webserver.cc:460] Webserver started at http://127.0.98.62:44593/ using document root <none> and password file <none>
I20260927 12:41:12.036628 6463 fs_manager.cc:359] Metadata directory not provided
I20260927 12:41:12.036684 6463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:41:12.036844 6463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:41:12.038825 6463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/master-0/data/instance:
uuid: "42be682b019e44ed8f1a42e5cf28499f"
format_stamp: "Formatted at 2026-09-27 12:41:12 on dist-test-slave-bnc1"
server_key: "167fa913263b35e59ec2dc592b2c9070"
server_key_iv: "7f7fac3059c6e2b728c72f523641b12b"
server_key_version: "encryptionkey@0"
I20260927 12:41:12.039341 6463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/master-0/wal/instance:
uuid: "42be682b019e44ed8f1a42e5cf28499f"
format_stamp: "Formatted at 2026-09-27 12:41:12 on dist-test-slave-bnc1"
server_key: "167fa913263b35e59ec2dc592b2c9070"
server_key_iv: "7f7fac3059c6e2b728c72f523641b12b"
server_key_version: "encryptionkey@0"
I20260927 12:41:12.042920 6463 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 12:41:12.045194 6478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:12.046449 6463 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 12:41:12.046586 6463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/master-0/wal
uuid: "42be682b019e44ed8f1a42e5cf28499f"
format_stamp: "Formatted at 2026-09-27 12:41:12 on dist-test-slave-bnc1"
server_key: "167fa913263b35e59ec2dc592b2c9070"
server_key_iv: "7f7fac3059c6e2b728c72f523641b12b"
server_key_version: "encryptionkey@0"
I20260927 12:41:12.046694 6463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:41:12.076383 6463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:41:12.079195 6463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:41:12.079396 6463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:41:12.086199 6463 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46303
I20260927 12:41:12.086467 6530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46303 every 8 connection(s)
I20260927 12:41:12.087142 6463 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:41:12.089937 6531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:41:12.095357 6531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42be682b019e44ed8f1a42e5cf28499f: Bootstrap starting.
I20260927 12:41:12.096436 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 6463
I20260927 12:41:12.096591 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/master-0/wal/instance
I20260927 12:41:12.096849 392 external_mini_cluster.cc:1428] Setting key 3c5583390c111fcfb4e8f6730106ba5a
I20260927 12:41:12.098073 6531 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42be682b019e44ed8f1a42e5cf28499f: Neither blocks nor log segments found. Creating new log.
I20260927 12:41:12.098943 6531 log.cc:824] T 00000000000000000000000000000000 P 42be682b019e44ed8f1a42e5cf28499f: Log is configured to *not* fsync() on all Append() calls
I20260927 12:41:12.101094 6531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42be682b019e44ed8f1a42e5cf28499f: No bootstrap required, opened a new log
I20260927 12:41:12.105180 6531 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42be682b019e44ed8f1a42e5cf28499f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42be682b019e44ed8f1a42e5cf28499f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46303 } }
I20260927 12:41:12.105399 6531 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42be682b019e44ed8f1a42e5cf28499f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:41:12.105435 6531 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42be682b019e44ed8f1a42e5cf28499f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42be682b019e44ed8f1a42e5cf28499f, State: Initialized, Role: FOLLOWER
I20260927 12:41:12.105911 6531 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42be682b019e44ed8f1a42e5cf28499f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42be682b019e44ed8f1a42e5cf28499f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46303 } }
I20260927 12:41:12.106073 6531 raft_consensus.cc:388] T 00000000000000000000000000000000 P 42be682b019e44ed8f1a42e5cf28499f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:41:12.106161 6531 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42be682b019e44ed8f1a42e5cf28499f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:41:12.106287 6531 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42be682b019e44ed8f1a42e5cf28499f [term 0 FOLLOWER]: Advancing to term 1
Sep 27 12:41:12 dist-test-slave-bnc1 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 12:41:12.107357 6531 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42be682b019e44ed8f1a42e5cf28499f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42be682b019e44ed8f1a42e5cf28499f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46303 } }
I20260927 12:41:12.107752 6531 leader_election.cc:302] T 00000000000000000000000000000000 P 42be682b019e44ed8f1a42e5cf28499f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 42be682b019e44ed8f1a42e5cf28499f; no voters:
I20260927 12:41:12.108142 6531 leader_election.cc:288] T 00000000000000000000000000000000 P 42be682b019e44ed8f1a42e5cf28499f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:41:12.108249 6536 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42be682b019e44ed8f1a42e5cf28499f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:41:12.108513 6536 raft_consensus.cc:686] T 00000000000000000000000000000000 P 42be682b019e44ed8f1a42e5cf28499f [term 1 LEADER]: Becoming Leader. State: Replica: 42be682b019e44ed8f1a42e5cf28499f, State: Running, Role: LEADER
I20260927 12:41:12.108840 6536 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42be682b019e44ed8f1a42e5cf28499f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42be682b019e44ed8f1a42e5cf28499f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46303 } }
I20260927 12:41:12.109608 6531 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42be682b019e44ed8f1a42e5cf28499f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:41:12.110739 6538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42be682b019e44ed8f1a42e5cf28499f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42be682b019e44ed8f1a42e5cf28499f. Latest consensus state: current_term: 1 leader_uuid: "42be682b019e44ed8f1a42e5cf28499f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42be682b019e44ed8f1a42e5cf28499f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46303 } } }
I20260927 12:41:12.110739 6537 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42be682b019e44ed8f1a42e5cf28499f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "42be682b019e44ed8f1a42e5cf28499f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42be682b019e44ed8f1a42e5cf28499f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46303 } } }
I20260927 12:41:12.110888 6538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42be682b019e44ed8f1a42e5cf28499f [sys.catalog]: This master's current role is: LEADER
I20260927 12:41:12.110888 6537 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42be682b019e44ed8f1a42e5cf28499f [sys.catalog]: This master's current role is: LEADER
I20260927 12:41:12.111269 6545 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:41:12.113911 6534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:12.098078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44580 (local address 127.0.98.62:46303)
0927 12:41:12.098536 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:12.098554 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:12.098593 (+ 39us) server_negotiation.cc:407] Connection header received
0927 12:41:12.099417 (+ 824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:12.099435 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:12.099800 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:12.100149 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:12.100847 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:12.102311 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:12.103287 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:12.104087 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:12.104265 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:12.107182 (+ 2917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:12.107192 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:12.107213 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:41:12.107254 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:12.109820 (+ 2566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:12.110389 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:12.110397 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:12.110403 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:12.110583 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:12.110948 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:12.110951 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:12.110953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:12.111542 (+ 589us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:12.112416 (+ 874us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:12.112575 (+ 159us) server_negotiation.cc:299] Negotiation successful
0927 12:41:12.112911 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":301,"thread_start_us":120,"threads_started":1}
I20260927 12:41:12.114470 6545 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:41:12.119513 6545 catalog_manager.cc:1347] Generated new cluster ID: d7aa23e71b254e6d8a29824dd7eff1f3
I20260927 12:41:12.119578 6545 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:41:12.145834 6545 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:41:12.146827 6545 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:41:12.152241 6545 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 42be682b019e44ed8f1a42e5cf28499f: Generated new TSK 0
I20260927 12:41:12.152871 6545 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260927 12:41:12.184326 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46303
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:41:12.292922 6557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:41:12.293219 6557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:41:12.293295 6557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:41:12.297343 6557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:41:12.297437 6557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:41:12.297531 6557 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:41:12.302793 6557 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6557
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:41:12.303884 6557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:41:12.304947 6557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:41:12.314502 6562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:12.314507 6563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:12.314679 6557 server_base.cc:1034] running on GCE node
W20260927 12:41:12.314513 6565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:12.315340 6557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:41:12.315980 6557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:41:12.317158 6557 hybrid_clock.cc:648] HybridClock initialized: now 1790512872317126 us; error 47 us; skew 500 ppm
Sep 27 12:41:12 dist-test-slave-bnc1 krb5kdc[6450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512872, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:41:12.320039 6557 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 12:41:12.320962 6557 webserver.cc:460] Webserver started at http://127.0.98.1:42085/ using document root <none> and password file <none>
I20260927 12:41:12.321535 6557 fs_manager.cc:359] Metadata directory not provided
I20260927 12:41:12.321683 6557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:41:12.321915 6557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:41:12.324215 6557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-0/data/instance:
uuid: "4ac28926eacd419bad1e9f8b5e063a36"
format_stamp: "Formatted at 2026-09-27 12:41:12 on dist-test-slave-bnc1"
server_key: "9db758d680f597c5fd5d7cbc6f7ded87"
server_key_iv: "25cbd11a48ae3e6062cf931fb26975f0"
server_key_version: "encryptionkey@0"
I20260927 12:41:12.324790 6557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4ac28926eacd419bad1e9f8b5e063a36"
format_stamp: "Formatted at 2026-09-27 12:41:12 on dist-test-slave-bnc1"
server_key: "9db758d680f597c5fd5d7cbc6f7ded87"
server_key_iv: "25cbd11a48ae3e6062cf931fb26975f0"
server_key_version: "encryptionkey@0"
I20260927 12:41:12.328440 6557 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 12:41:12.330878 6572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:12.332113 6557 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 12:41:12.332237 6557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-0/wal
uuid: "4ac28926eacd419bad1e9f8b5e063a36"
format_stamp: "Formatted at 2026-09-27 12:41:12 on dist-test-slave-bnc1"
server_key: "9db758d680f597c5fd5d7cbc6f7ded87"
server_key_iv: "25cbd11a48ae3e6062cf931fb26975f0"
server_key_version: "encryptionkey@0"
I20260927 12:41:12.332338 6557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:41:12.355201 6557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:41:12.358244 6557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:41:12.358449 6557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:41:12.359087 6557 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:41:12.359900 6557 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:41:12.359968 6557 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:12.360040 6557 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:41:12.360081 6557 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:12.369264 6557 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42501
I20260927 12:41:12.369295 6685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42501 every 8 connection(s)
I20260927 12:41:12.370204 6557 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:41:12.370548 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 6557
I20260927 12:41:12.370680 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-0/wal/instance
I20260927 12:41:12.371001 392 external_mini_cluster.cc:1428] Setting key b79d72fcaadfbdefd77756964557c7ad
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Sep 27 12:41:12 dist-test-slave-bnc1 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512872, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
I20260927 12:41:12.385201 6534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:12.372214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46031 (local address 127.0.98.62:46303)
0927 12:41:12.372359 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:12.372365 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:12.373395 (+ 1030us) server_negotiation.cc:407] Connection header received
0927 12:41:12.375033 (+ 1638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:12.375039 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:12.375114 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:12.375214 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:12.376358 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:12.377258 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:12.378470 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:12.378850 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:12.378968 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:12.381478 (+ 2510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:12.381482 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:12.381488 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:41:12.381517 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:12.383173 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:12.383674 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:12.383679 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:12.383681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:12.383765 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:12.384098 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:12.384103 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:12.384111 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:12.384294 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:12.384400 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:12.384905 (+ 505us) server_negotiation.cc:299] Negotiation successful
0927 12:41:12.385028 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 12:41:12.386096 6688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:12.372696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46303 (local address 127.0.98.1:46031)
0927 12:41:12.373218 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:12.373278 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:12.374819 (+ 1541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:12.375364 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:12.375371 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:12.375733 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:12.376208 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:12.376220 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:12.377390 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:12.377393 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:12.378349 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:12.378358 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:12.379002 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:12.379005 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:12.379142 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:12.379753 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:12.379769 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:12.381312 (+ 1543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:41:12.383291 (+ 1979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:12.383296 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:12.383313 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:12.383570 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:12.383859 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:12.383862 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:12.383865 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:12.384005 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:12.384399 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:41:12.384405 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:41:12.384673 (+ 268us) client_negotiation.cc:769] Sending connection context
0927 12:41:12.384866 (+ 193us) client_negotiation.cc:240] Negotiation successful
0927 12:41:12.385197 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":120,"threads_started":1}
I20260927 12:41:12.387360 6686 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46303
I20260927 12:41:12.387603 6686 heartbeater.cc:464] Registering TS with master...
I20260927 12:41:12.388082 6686 heartbeater.cc:511] Master 127.0.98.62:46303 requested a full tablet report, sending...
I20260927 12:41:12.389737 6495 ts_manager.cc:194] Registered new tserver with Master: 4ac28926eacd419bad1e9f8b5e063a36 (127.0.98.1:42501)
I20260927 12:41:12.391263 6495 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46031
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260927 12:41:12.400162 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46303
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:41:12.508986 6691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:41:12.509274 6691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:41:12.509317 6691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:41:12.513813 6691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:41:12.513870 6691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:41:12.513931 6691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:41:12.519158 6691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:41:12.520134 6691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:41:12.521078 6691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:41:12.530828 6699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:12.530915 6697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:12.530866 6696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:12.530979 6691 server_base.cc:1034] running on GCE node
I20260927 12:41:12.531472 6691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:41:12.532158 6691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:41:12.533324 6691 hybrid_clock.cc:648] HybridClock initialized: now 1790512872533289 us; error 44 us; skew 500 ppm
Sep 27 12:41:12 dist-test-slave-bnc1 krb5kdc[6450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512872, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:41:12.540016 6691 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 12:41:12.540874 6691 webserver.cc:460] Webserver started at http://127.0.98.2:33365/ using document root <none> and password file <none>
I20260927 12:41:12.541429 6691 fs_manager.cc:359] Metadata directory not provided
I20260927 12:41:12.541517 6691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:41:12.541757 6691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:41:12.543782 6691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-1/data/instance:
uuid: "9fd3b55ca4154ec69ee88b69c88fb053"
format_stamp: "Formatted at 2026-09-27 12:41:12 on dist-test-slave-bnc1"
server_key: "69624e0f644ed3c6f23750ac607b790d"
server_key_iv: "994cdef9d764189eb8a54613c9c968a1"
server_key_version: "encryptionkey@0"
I20260927 12:41:12.544337 6691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9fd3b55ca4154ec69ee88b69c88fb053"
format_stamp: "Formatted at 2026-09-27 12:41:12 on dist-test-slave-bnc1"
server_key: "69624e0f644ed3c6f23750ac607b790d"
server_key_iv: "994cdef9d764189eb8a54613c9c968a1"
server_key_version: "encryptionkey@0"
I20260927 12:41:12.547971 6691 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 12:41:12.550513 6706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:12.551746 6691 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 12:41:12.551872 6691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-1/wal
uuid: "9fd3b55ca4154ec69ee88b69c88fb053"
format_stamp: "Formatted at 2026-09-27 12:41:12 on dist-test-slave-bnc1"
server_key: "69624e0f644ed3c6f23750ac607b790d"
server_key_iv: "994cdef9d764189eb8a54613c9c968a1"
server_key_version: "encryptionkey@0"
I20260927 12:41:12.551976 6691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:41:12.566426 6691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:41:12.569693 6691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:41:12.569890 6691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:41:12.570482 6691 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:41:12.571332 6691 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:41:12.571403 6691 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:12.571475 6691 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:41:12.571517 6691 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:12.580363 6691 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43043
I20260927 12:41:12.580389 6819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43043 every 8 connection(s)
I20260927 12:41:12.581251 6691 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:41:12.586191 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 6691
I20260927 12:41:12.586287 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-1/wal/instance
I20260927 12:41:12.586549 392 external_mini_cluster.cc:1428] Setting key 434864254e64f9ecd81d7a864a515327
Sep 27 12:41:12 dist-test-slave-bnc1 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512872, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 12:41:12.595161 6534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:12.583124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58513 (local address 127.0.98.62:46303)
0927 12:41:12.583253 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:12.583265 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:12.584000 (+ 735us) server_negotiation.cc:407] Connection header received
0927 12:41:12.585258 (+ 1258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:12.585260 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:12.585304 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:12.585378 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:12.586628 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:12.587162 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:12.588322 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:12.588545 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:12.588663 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:12.591242 (+ 2579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:12.591248 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:12.591254 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:41:12.591288 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:12.592999 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:12.593512 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:12.593517 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:12.593520 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:12.593653 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:12.593984 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:12.593989 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:12.593994 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:12.594243 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:12.594352 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:12.594827 (+ 475us) server_negotiation.cc:299] Negotiation successful
0927 12:41:12.594963 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 12:41:12.595901 6822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:12.583441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46303 (local address 127.0.98.2:58513)
0927 12:41:12.583851 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:12.583891 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:12.585070 (+ 1179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:12.585533 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:12.585540 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:12.585906 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:12.586464 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:12.586480 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:12.587299 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:12.587302 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:12.588189 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:12.588198 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:12.588697 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:12.588700 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:12.588882 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:12.589499 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:12.589514 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:12.591058 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:41:12.593145 (+ 2087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:12.593149 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:12.593162 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:12.593407 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:12.593748 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:12.593751 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:12.593752 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:12.593884 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:12.594347 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:41:12.594352 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:41:12.594578 (+ 226us) client_negotiation.cc:769] Sending connection context
0927 12:41:12.594741 (+ 163us) client_negotiation.cc:240] Negotiation successful
0927 12:41:12.594986 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":106,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 12:41:12.597177 6820 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46303
I20260927 12:41:12.597432 6820 heartbeater.cc:464] Registering TS with master...
I20260927 12:41:12.597949 6820 heartbeater.cc:511] Master 127.0.98.62:46303 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
I20260927 12:41:12.599040 6495 ts_manager.cc:194] Registered new tserver with Master: 9fd3b55ca4154ec69ee88b69c88fb053 (127.0.98.2:43043)
I20260927 12:41:12.599581 6495 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58513
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260927 12:41:12.617089 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46303
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:41:12.726284 6825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:41:12.726552 6825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:41:12.726593 6825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:41:12.730788 6825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:41:12.730854 6825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:41:12.730911 6825 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:41:12.736222 6825 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6825
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:41:12.737262 6825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:41:12.738310 6825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:41:12.747776 6833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:12.747826 6825 server_base.cc:1034] running on GCE node
W20260927 12:41:12.747754 6830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:12.747758 6831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:12.748497 6825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:41:12.749120 6825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:41:12.750314 6825 hybrid_clock.cc:648] HybridClock initialized: now 1790512872750298 us; error 30 us; skew 500 ppm
Sep 27 12:41:12 dist-test-slave-bnc1 krb5kdc[6450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512872, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:41:12.753098 6825 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 12:41:12.754015 6825 webserver.cc:460] Webserver started at http://127.0.98.3:45313/ using document root <none> and password file <none>
I20260927 12:41:12.754572 6825 fs_manager.cc:359] Metadata directory not provided
I20260927 12:41:12.754663 6825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:41:12.754987 6825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:41:12.757325 6825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-2/data/instance:
uuid: "849bb514fb834ec38a1f428e351782d0"
format_stamp: "Formatted at 2026-09-27 12:41:12 on dist-test-slave-bnc1"
server_key: "4d0511ff1c623af551429d3b25c2084d"
server_key_iv: "06f2ad7ac3969b37998aeba6cc1ed363"
server_key_version: "encryptionkey@0"
I20260927 12:41:12.757941 6825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-2/wal/instance:
uuid: "849bb514fb834ec38a1f428e351782d0"
format_stamp: "Formatted at 2026-09-27 12:41:12 on dist-test-slave-bnc1"
server_key: "4d0511ff1c623af551429d3b25c2084d"
server_key_iv: "06f2ad7ac3969b37998aeba6cc1ed363"
server_key_version: "encryptionkey@0"
I20260927 12:41:12.761770 6825 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 12:41:12.764081 6840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:12.765267 6825 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:41:12.765406 6825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-2/wal
uuid: "849bb514fb834ec38a1f428e351782d0"
format_stamp: "Formatted at 2026-09-27 12:41:12 on dist-test-slave-bnc1"
server_key: "4d0511ff1c623af551429d3b25c2084d"
server_key_iv: "06f2ad7ac3969b37998aeba6cc1ed363"
server_key_version: "encryptionkey@0"
I20260927 12:41:12.765519 6825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:41:12.790539 6825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:41:12.793404 6825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:41:12.793640 6825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:41:12.794255 6825 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:41:12.795077 6825 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:41:12.795150 6825 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:12.795229 6825 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:41:12.795265 6825 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:12.804302 6825 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40415
I20260927 12:41:12.804339 6953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40415 every 8 connection(s)
I20260927 12:41:12.805231 6825 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:41:12.813474 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 6825
I20260927 12:41:12.813630 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-2/wal/instance
I20260927 12:41:12.813910 392 external_mini_cluster.cc:1428] Setting key 672f3bd5364810df7b68b7110fe82267
Sep 27 12:41:12 dist-test-slave-bnc1 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512872, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 12:41:12.819140 6534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:12.807199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56697 (local address 127.0.98.62:46303)
0927 12:41:12.807375 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:12.807380 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:12.808155 (+ 775us) server_negotiation.cc:407] Connection header received
0927 12:41:12.809583 (+ 1428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:12.809586 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:12.809633 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:12.809713 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:12.810973 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:12.811546 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:12.812613 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:12.812832 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:12.812909 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:12.815544 (+ 2635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:12.815547 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:12.815549 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:41:12.815572 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:12.817045 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:12.817597 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:12.817600 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:12.817602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:12.817690 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:12.818049 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:12.818052 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:12.818054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:12.818253 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:12.818361 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:12.818863 (+ 502us) server_negotiation.cc:299] Negotiation successful
0927 12:41:12.818982 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 12:41:12.820009 6956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:12.807533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46303 (local address 127.0.98.3:56697)
0927 12:41:12.807994 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:12.808036 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:12.809333 (+ 1297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:12.809871 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:12.809878 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:12.810283 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:12.810841 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:12.810855 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:12.811658 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:12.811661 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:12.812513 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:12.812521 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:12.812954 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:12.812957 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:12.813092 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:12.813790 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:12.813807 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:12.815369 (+ 1562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:41:12.817169 (+ 1800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:12.817176 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:12.817192 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:12.817455 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:12.817784 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:12.817788 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:12.817791 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:12.817944 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:12.818359 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:41:12.818366 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:41:12.818621 (+ 255us) client_negotiation.cc:769] Sending connection context
0927 12:41:12.818824 (+ 203us) client_negotiation.cc:240] Negotiation successful
0927 12:41:12.819158 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":115,"threads_started":1}
I20260927 12:41:12.821146 6954 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46303
I20260927 12:41:12.821399 6954 heartbeater.cc:464] Registering TS with master...
I20260927 12:41:12.821985 6954 heartbeater.cc:511] Master 127.0.98.62:46303 requested a full tablet report, sending...
I20260927 12:41:12.823040 6495 ts_manager.cc:194] Registered new tserver with Master: 849bb514fb834ec38a1f428e351782d0 (127.0.98.3:40415)
I20260927 12:41:12.823612 6495 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56697
I20260927 12:41:12.827946 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260927 12:41:12.854722 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:46395
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46303,127.0.98.61:46395
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:41:12.967952 6959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:41:12.968276 6959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:41:12.968350 6959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:41:12.972564 6959 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:41:12.972664 6959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:41:12.972702 6959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:41:12.972739 6959 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:41:12.972769 6959 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:41:12.978365 6959 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46303,127.0.98.61:46395
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:46395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6959
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
W20260927 12:41:12.978643 6959 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:46303,127.0.98.61:46395'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260927 12:41:12.979580 6959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:41:12.980620 6959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:41:12.989645 6964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:12.989614 6967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:12.989661 6965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:12.989813 6959 server_base.cc:1034] running on GCE node
I20260927 12:41:12.990540 6959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:41:12.991720 6959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:41:12.992890 6959 hybrid_clock.cc:648] HybridClock initialized: now 1790512872992861 us; error 40 us; skew 500 ppm
Sep 27 12:41:12 dist-test-slave-bnc1 krb5kdc[6450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512872, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:41:12.995724 6959 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260927 12:41:12.996635 6959 webserver.cc:460] Webserver started at http://127.0.98.61:38709/ using document root <none> and password file <none>
I20260927 12:41:12.997193 6959 fs_manager.cc:359] Metadata directory not provided
I20260927 12:41:12.997288 6959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:41:12.997550 6959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:41:12.999681 6959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/master-1/data/instance:
uuid: "04babc8b97154b25ac1e359afb91c667"
format_stamp: "Formatted at 2026-09-27 12:41:12 on dist-test-slave-bnc1"
server_key: "5def0e7e95958e616f53dec4c94bfc1b"
server_key_iv: "a0629928575c8059d62a0fbda496f343"
server_key_version: "encryptionkey@0"
I20260927 12:41:13.000257 6959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/master-1/wal/instance:
uuid: "04babc8b97154b25ac1e359afb91c667"
format_stamp: "Formatted at 2026-09-27 12:41:12 on dist-test-slave-bnc1"
server_key: "5def0e7e95958e616f53dec4c94bfc1b"
server_key_iv: "a0629928575c8059d62a0fbda496f343"
server_key_version: "encryptionkey@0"
I20260927 12:41:13.004055 6959 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260927 12:41:13.006567 6974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:13.007815 6959 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 12:41:13.007956 6959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/master-1/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/master-1/wal
uuid: "04babc8b97154b25ac1e359afb91c667"
format_stamp: "Formatted at 2026-09-27 12:41:12 on dist-test-slave-bnc1"
server_key: "5def0e7e95958e616f53dec4c94bfc1b"
server_key_iv: "a0629928575c8059d62a0fbda496f343"
server_key_version: "encryptionkey@0"
I20260927 12:41:13.008067 6959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:41:13.021871 6959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:41:13.024726 6959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:41:13.024950 6959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:41:13.032094 6959 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:46395
I20260927 12:41:13.032369 7026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:46395 every 8 connection(s)
I20260927 12:41:13.033142 6959 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/master-1/data/info.pb
I20260927 12:41:13.035933 7027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:41:13.039641 7027 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46303 } has no permanent_uuid. Determining permanent_uuid...
I20260927 12:41:13.042376 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 6959
Sep 27 12:41:13 dist-test-slave-bnc1 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512872, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 27 12:41:13 dist-test-slave-bnc1 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 12:41:13.054662 6534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:13.041423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44582 (local address 127.0.98.62:46303)
0927 12:41:13.041562 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:13.041604 (+ 42us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:13.042549 (+ 945us) server_negotiation.cc:407] Connection header received
0927 12:41:13.043639 (+ 1090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:13.043659 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:13.043721 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:13.043834 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:13.045017 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.045894 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.047183 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.047463 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.047580 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:13.050587 (+ 3007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:13.050590 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:13.050592 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:41:13.050617 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:13.052302 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:13.052793 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:13.052799 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:13.052801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:13.052890 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:13.053301 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:13.053303 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:13.053305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:13.053647 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:13.053727 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:13.054337 (+ 610us) server_negotiation.cc:299] Negotiation successful
0927 12:41:13.054490 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 12:41:13.055459 7031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:13.043271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55760 (local address 127.0.98.61:46395)
0927 12:41:13.043739 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:13.043753 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:13.043782 (+ 29us) server_negotiation.cc:407] Connection header received
0927 12:41:13.043925 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:13.043931 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:13.044117 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:13.044276 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:13.045197 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.046342 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.047383 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.047930 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.048084 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:13.051115 (+ 3031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:13.051124 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:13.051138 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:41:13.051167 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:13.052959 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:13.053450 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:13.053455 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:13.053457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:13.053531 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:13.053881 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:13.053883 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:13.053884 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:13.054093 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:13.054272 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:13.054494 (+ 222us) server_negotiation.cc:299] Negotiation successful
0927 12:41:13.054656 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":66,"server-negotiator.queue_time_us":368,"thread_start_us":128,"threads_started":1}
I20260927 12:41:13.055466 7029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:13.041845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46303 (local address 127.0.0.1:44582)
0927 12:41:13.042381 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:13.042438 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:13.043333 (+ 895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:13.043942 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:13.043949 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:13.044328 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:13.044833 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:13.044851 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.046020 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.046023 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:13.047052 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:13.047061 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.047607 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.047610 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:13.047818 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:13.048479 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:13.048502 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:13.050428 (+ 1926us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:41:13.052410 (+ 1982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:13.052417 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:13.052424 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:13.052685 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:13.052983 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:13.052985 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:13.052986 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:13.053201 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:13.053747 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:41:13.053753 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:41:13.054033 (+ 280us) client_negotiation.cc:769] Sending connection context
0927 12:41:13.054271 (+ 238us) client_negotiation.cc:240] Negotiation successful
0927 12:41:13.054597 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"mutex_wait_us":55,"thread_start_us":113,"threads_started":1}
I20260927 12:41:13.056874 7027 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46395 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 12:41:13 dist-test-slave-bnc1 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512872, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 12:41:13.064299 7029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:13.057420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46395 (local address 127.0.0.1:55774)
0927 12:41:13.057591 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:13.057605 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:13.057681 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:13.057909 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:13.057911 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:13.058067 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:13.058142 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:13.058147 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.058852 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.058858 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:13.059359 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:13.059366 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.059815 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.059818 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:13.059908 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:13.060427 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:13.060437 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:13.061616 (+ 1179us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:41:13.063112 (+ 1496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:13.063115 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:13.063117 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:13.063310 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:13.063545 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:13.063547 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:13.063548 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:13.063619 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:13.063916 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:41:13.063918 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:41:13.063988 (+ 70us) client_negotiation.cc:769] Sending connection context
0927 12:41:13.064069 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 12:41:13.064172 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92}
I20260927 12:41:13.064325 7031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:13.057487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55774 (local address 127.0.98.61:46395)
0927 12:41:13.057592 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:13.057598 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:13.057645 (+ 47us) server_negotiation.cc:407] Connection header received
0927 12:41:13.057776 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:13.057780 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:13.057813 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:13.057902 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:13.058250 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.058752 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.059463 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.059715 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.059819 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:13.061733 (+ 1914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:13.061736 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:13.061738 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:41:13.061757 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:13.063015 (+ 1258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:13.063404 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:13.063406 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:13.063408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:13.063470 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:13.063689 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:13.063692 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:13.063693 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:13.063837 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:13.063908 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:13.064118 (+ 210us) server_negotiation.cc:299] Negotiation successful
0927 12:41:13.064220 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":16,"server-negotiator.queue_time_us":29}
I20260927 12:41:13.067569 7027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667: Bootstrap starting.
I20260927 12:41:13.069980 7027 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667: Neither blocks nor log segments found. Creating new log.
I20260927 12:41:13.070698 7027 log.cc:824] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667: Log is configured to *not* fsync() on all Append() calls
I20260927 12:41:13.072538 7027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667: No bootstrap required, opened a new log
I20260927 12:41:13.075474 7027 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42be682b019e44ed8f1a42e5cf28499f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46303 } } peers { permanent_uuid: "04babc8b97154b25ac1e359afb91c667" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46395 } }
I20260927 12:41:13.075699 7027 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:41:13.075778 7027 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04babc8b97154b25ac1e359afb91c667, State: Initialized, Role: FOLLOWER
I20260927 12:41:13.076200 7027 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42be682b019e44ed8f1a42e5cf28499f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46303 } } peers { permanent_uuid: "04babc8b97154b25ac1e359afb91c667" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46395 } }
I20260927 12:41:13.076660 7033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42be682b019e44ed8f1a42e5cf28499f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46303 } } peers { permanent_uuid: "04babc8b97154b25ac1e359afb91c667" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46395 } } }
I20260927 12:41:13.076776 7033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 12:41:13.077328 7027 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 12:41:13.082396 7044 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 12:41:13.082504 7044 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 12:41:13.084136 6959 master_runner.cc:428] Detected that this master 04babc8b97154b25ac1e359afb91c667 is joining an existing cluster
I20260927 12:41:13.084216 6959 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:46395
I20260927 12:41:13.094679 7031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:13.086807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55784 (local address 127.0.98.61:46395)
0927 12:41:13.086904 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:13.086910 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:13.086923 (+ 13us) server_negotiation.cc:407] Connection header received
0927 12:41:13.086958 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:13.086960 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:13.086991 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:13.087076 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:13.087603 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.088439 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.089598 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.089942 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.090014 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:13.091432 (+ 1418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:13.091436 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:13.091439 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:41:13.091460 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:13.093092 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:13.093519 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:13.093522 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:13.093524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:13.093617 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:13.093878 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:13.093880 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:13.093882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:13.094062 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:13.094127 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:13.094403 (+ 276us) server_negotiation.cc:299] Negotiation successful
0927 12:41:13.094504 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":29,"spinlock_wait_cycles":896}
I20260927 12:41:13.095150 7051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:13.086275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46395 (local address 127.0.0.1:55784)
0927 12:41:13.086721 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:13.086735 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:13.086823 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:13.087100 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:13.087103 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:13.087327 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:13.087485 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:13.087493 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.088553 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.088556 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:13.089448 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:13.089457 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.090151 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.090154 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:13.090240 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:13.090760 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:13.090776 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:13.091291 (+ 515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:41:13.093199 (+ 1908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:13.093202 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:13.093204 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:13.093423 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:13.093710 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:13.093713 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:13.093715 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:13.093787 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:13.094182 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:41:13.094185 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:41:13.094251 (+ 66us) client_negotiation.cc:769] Sending connection context
0927 12:41:13.094877 (+ 626us) client_negotiation.cc:240] Negotiation successful
0927 12:41:13.094996 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":100,"threads_started":1}
I20260927 12:41:13.095752 6534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:13.086603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44586 (local address 127.0.98.62:46303)
0927 12:41:13.086717 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:13.086722 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:13.087021 (+ 299us) server_negotiation.cc:407] Connection header received
0927 12:41:13.087173 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:13.087176 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:13.087218 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:13.087266 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:13.087900 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.088712 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.089928 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.090252 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.090327 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:13.092164 (+ 1837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:13.092167 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:13.092171 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:41:13.092433 (+ 262us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:13.094035 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:13.094497 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:13.094500 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:13.094502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:13.094574 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:13.094905 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:13.094908 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:13.094909 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:13.095096 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:13.095163 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:13.095515 (+ 352us) server_negotiation.cc:299] Negotiation successful
0927 12:41:13.095622 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 12:41:13.096236 7050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:13.086340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46303 (local address 127.0.0.1:44586)
0927 12:41:13.086945 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:13.086958 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:13.087078 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:13.087409 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:13.087411 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:13.087610 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:13.087768 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:13.087775 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.088850 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.088853 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:13.089803 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:13.089812 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.090778 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.090782 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:13.090873 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:13.091345 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:13.091358 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:13.092023 (+ 665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:41:13.094159 (+ 2136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:13.094162 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:13.094164 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:13.094395 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:13.094675 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:13.094678 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:13.094680 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:13.094791 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:13.095234 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:41:13.095237 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:41:13.095330 (+ 93us) client_negotiation.cc:769] Sending connection context
0927 12:41:13.095931 (+ 601us) client_negotiation.cc:240] Negotiation successful
0927 12:41:13.096064 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"mutex_wait_us":4,"thread_start_us":83,"threads_started":1}
Sep 27 12:41:13 dist-test-slave-bnc1 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512872, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 12:41:13.110772 7031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:13.100829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55796 (local address 127.0.98.61:46395)
0927 12:41:13.100963 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:13.100967 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:13.101978 (+ 1011us) server_negotiation.cc:407] Connection header received
0927 12:41:13.102153 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:13.102156 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:13.102204 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:13.102292 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:13.103006 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.103682 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.104876 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.105087 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.105164 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:13.107426 (+ 2262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:13.107432 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:13.107434 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:41:13.107459 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:13.109016 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:13.109475 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:13.109481 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:13.109483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:13.109562 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:13.109898 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:13.109901 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:13.109903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:13.110066 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:13.110169 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:13.110515 (+ 346us) server_negotiation.cc:299] Negotiation successful
0927 12:41:13.110623 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 12:41:13.110817 7052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:13.100730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46395 (local address 127.0.0.1:55796)
0927 12:41:13.101841 (+ 1111us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:13.101898 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:13.102024 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:13.102331 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:13.102337 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:13.102755 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:13.102881 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:13.102889 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.103796 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.103799 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:13.104760 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:13.104770 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.105190 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.105194 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:13.105297 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:13.105816 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:13.105830 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:13.107293 (+ 1463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:41:13.109120 (+ 1827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:13.109130 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:13.109136 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:13.109374 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:13.109682 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:13.109686 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:13.109688 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:13.109773 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:13.110183 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:41:13.110191 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:41:13.110320 (+ 129us) client_negotiation.cc:769] Sending connection context
0927 12:41:13.110467 (+ 147us) client_negotiation.cc:240] Negotiation successful
0927 12:41:13.110646 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":929,"thread_start_us":65,"threads_started":1}
I20260927 12:41:13.112447 6495 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:46395: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "04babc8b97154b25ac1e359afb91c667"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 46395
}
attrs {
promote: true
}
}
dest_uuid: "42be682b019e44ed8f1a42e5cf28499f"
cas_config_opid_index: -1
I20260927 12:41:13.113085 6495 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42be682b019e44ed8f1a42e5cf28499f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "42be682b019e44ed8f1a42e5cf28499f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46303 } } peers { permanent_uuid: "04babc8b97154b25ac1e359afb91c667" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46395 } attrs { promote: true } }
I20260927 12:41:13.115013 7055 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 42be682b019e44ed8f1a42e5cf28499f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 04babc8b97154b25ac1e359afb91c667 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "42be682b019e44ed8f1a42e5cf28499f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46303 } } peers { permanent_uuid: "04babc8b97154b25ac1e359afb91c667" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46395 } attrs { promote: true } } }
I20260927 12:41:13.115602 7001 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:41:13.116482 7055 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:46395
I20260927 12:41:13.116847 6959 master.cc:537] Master@127.0.98.61:46395 shutting down...
I20260927 12:41:13.116873 7055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42be682b019e44ed8f1a42e5cf28499f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "42be682b019e44ed8f1a42e5cf28499f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "42be682b019e44ed8f1a42e5cf28499f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46303 } } peers { permanent_uuid: "04babc8b97154b25ac1e359afb91c667" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46395 } attrs { promote: true } } }
I20260927 12:41:13.116956 7055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42be682b019e44ed8f1a42e5cf28499f [sys.catalog]: This master's current role is: LEADER
I20260927 12:41:13.117129 7001 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667 [term 1 FOLLOWER]: Refusing update from remote peer 42be682b019e44ed8f1a42e5cf28499f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260927 12:41:13.117849 7054 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 42be682b019e44ed8f1a42e5cf28499f [LEADER]: Connected to new peer: Peer: permanent_uuid: "04babc8b97154b25ac1e359afb91c667" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46395 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
W20260927 12:41:13.120000 6481 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260927 12:41:13.121701 6959 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:41:13.121974 6959 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667 [term 1 FOLLOWER]: Raft consensus is shut down!
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
I20260927 12:41:13.122103 6959 tablet_replica.cc:330] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667: stopping tablet replica
I20260927 12:41:13.126264 7052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:13.126026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46395 (local address 127.0.0.1:55808)
0927 12:41:13.126146 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:13.126200 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 12:41:13.126410 6481 consensus_peers.cc:487] T 00000000000000000000000000000000 P 42be682b019e44ed8f1a42e5cf28499f -> Peer 04babc8b97154b25ac1e359afb91c667 (127.0.98.61:46395): Couldn't send request to peer 04babc8b97154b25ac1e359afb91c667. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46395: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 12:41:13.136766 6959 master.cc:552] Master@127.0.98.61:46395 shutdown complete.
I20260927 12:41:13.136818 6959 master_runner.cc:305] Clearing existing system tablet
I20260927 12:41:13.137085 6959 fs_manager.cc:359] Metadata directory not provided
I20260927 12:41:13.137156 6959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:41:13.138487 6959 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260927 12:41:13.139894 7060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:13.140113 6959 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:41:13.140259 6959 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 12:41:13.140306 6959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/master-1/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/master-1/wal
uuid: "04babc8b97154b25ac1e359afb91c667"
format_stamp: "Formatted at 2026-09-27 12:41:12 on dist-test-slave-bnc1"
server_key: "5def0e7e95958e616f53dec4c94bfc1b"
server_key_iv: "a0629928575c8059d62a0fbda496f343"
server_key_version: "encryptionkey@0"
I20260927 12:41:13.140719 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:42401
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46303,127.0.98.61:46395,127.0.98.60:42401
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 12:41:13.142382 6959 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:41:13.144678 6959 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 12:41:13.144754 6959 log.cc:1196] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667: Deleting WAL directory at /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260927 12:41:13.145061 6959 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667: Deleting consensus metadata
I20260927 12:41:13.145233 6959 master_runner.cc:315] Copying system tablet from 127.0.98.62:46303
I20260927 12:41:13.146355 6959 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:46303
I20260927 12:41:13.154959 6534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:13.147001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44602 (local address 127.0.98.62:46303)
0927 12:41:13.147130 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:13.147135 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:13.147350 (+ 215us) server_negotiation.cc:407] Connection header received
0927 12:41:13.147509 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:13.147512 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:13.147561 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:13.147653 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:13.148080 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.148607 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.149440 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.149802 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.149917 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:13.151700 (+ 1783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:13.151703 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:13.151706 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:41:13.151732 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:13.153222 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:13.153705 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:13.153709 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:13.153711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:13.153794 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:13.154080 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:13.154083 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:13.154085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:13.154262 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:13.154376 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:13.154677 (+ 301us) server_negotiation.cc:299] Negotiation successful
0927 12:41:13.154792 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 12:41:13.154969 7067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:13.146925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46303 (local address 127.0.0.1:44602)
0927 12:41:13.147248 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:13.147260 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:13.147399 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:13.147669 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:13.147672 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:13.147871 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:13.147960 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:13.147966 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.148714 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.148720 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:13.149316 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:13.149323 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.150032 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.150035 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:13.150127 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:13.151193 (+ 1066us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:13.151223 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:13.151564 (+ 341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:41:13.153362 (+ 1798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:13.153366 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:13.153368 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:13.153621 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:13.153889 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:13.153893 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:13.153895 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:13.153982 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:13.154377 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:41:13.154381 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:41:13.154498 (+ 117us) client_negotiation.cc:769] Sending connection context
0927 12:41:13.154607 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 12:41:13.154757 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":81,"threads_started":1}
I20260927 12:41:13.155531 6515 tablet_copy_service.cc:140] P 42be682b019e44ed8f1a42e5cf28499f: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 04babc8b97154b25ac1e359afb91c667 ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:44602)
I20260927 12:41:13.155709 6515 tablet_copy_service.cc:161] P 42be682b019e44ed8f1a42e5cf28499f: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 04babc8b97154b25ac1e359afb91c667 at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:44602: session id = 04babc8b97154b25ac1e359afb91c667-00000000000000000000000000000000
I20260927 12:41:13.157229 6515 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 42be682b019e44ed8f1a42e5cf28499f: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 12:41:13.158769 6959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 12:41:13.160674 6959 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667: tablet copy: Starting download of 0 data blocks...
I20260927 12:41:13.160869 6959 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667: tablet copy: Starting download of 1 WAL segments...
I20260927 12:41:13.162452 6959 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 12:41:13.163975 6515 tablet_copy_service.cc:342] P 42be682b019e44ed8f1a42e5cf28499f: Request end of tablet copy session 04babc8b97154b25ac1e359afb91c667-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:44602
I20260927 12:41:13.164114 6515 tablet_copy_service.cc:433] P 42be682b019e44ed8f1a42e5cf28499f: ending tablet copy session 04babc8b97154b25ac1e359afb91c667-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 04babc8b97154b25ac1e359afb91c667
I20260927 12:41:13.171137 6959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:41:13.172768 7068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:13.172820 7069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:13.173100 6959 server_base.cc:1034] running on GCE node
W20260927 12:41:13.173357 7071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:13.173599 6959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:41:13.173916 6959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:41:13.175055 6959 hybrid_clock.cc:648] HybridClock initialized: now 1790512873175071 us; error 48 us; skew 500 ppm
Sep 27 12:41:13 dist-test-slave-bnc1 krb5kdc[6450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512873, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:41:13.176571 6959 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260927 12:41:13.177052 6959 webserver.cc:460] Webserver started at http://127.0.98.61:42945/ using document root <none> and password file <none>
I20260927 12:41:13.177229 6959 fs_manager.cc:359] Metadata directory not provided
I20260927 12:41:13.177280 6959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:41:13.178433 6959 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260927 12:41:13.179297 7078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:13.179519 6959 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 12:41:13.179610 6959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/master-1/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/master-1/wal
uuid: "04babc8b97154b25ac1e359afb91c667"
format_stamp: "Formatted at 2026-09-27 12:41:12 on dist-test-slave-bnc1"
server_key: "5def0e7e95958e616f53dec4c94bfc1b"
server_key_iv: "a0629928575c8059d62a0fbda496f343"
server_key_version: "encryptionkey@0"
I20260927 12:41:13.179688 6959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:41:13.186239 6959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:41:13.188135 6959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:41:13.193272 6959 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:46395
I20260927 12:41:13.193770 7130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:46395 every 8 connection(s)
I20260927 12:41:13.193953 6959 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/master-1/data/info.pb
I20260927 12:41:13.195349 7131 sys_catalog.cc:263] Verifying existing consensus state
I20260927 12:41:13.196224 7131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667: Bootstrap starting.
I20260927 12:41:13.204080 7131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 12:41:13.204398 7131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667: Bootstrap complete.
I20260927 12:41:13.204887 7131 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "42be682b019e44ed8f1a42e5cf28499f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46303 } } peers { permanent_uuid: "04babc8b97154b25ac1e359afb91c667" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46395 } attrs { promote: true } }
I20260927 12:41:13.205035 7131 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 04babc8b97154b25ac1e359afb91c667, State: Initialized, Role: LEARNER
I20260927 12:41:13.205150 7131 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "42be682b019e44ed8f1a42e5cf28499f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46303 } } peers { permanent_uuid: "04babc8b97154b25ac1e359afb91c667" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46395 } attrs { promote: true } }
I20260927 12:41:13.205510 7134 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "42be682b019e44ed8f1a42e5cf28499f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46303 } } peers { permanent_uuid: "04babc8b97154b25ac1e359afb91c667" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46395 } attrs { promote: true } } }
I20260927 12:41:13.205699 7131 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:41:13.205705 7134 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667 [sys.catalog]: This master's current role is: LEARNER
I20260927 12:41:13.208420 7145 catalog_manager.cc:1259] Loaded cluster ID: d7aa23e71b254e6d8a29824dd7eff1f3
I20260927 12:41:13.208526 7145 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667: loading cluster ID for follower catalog manager: success
I20260927 12:41:13.210695 7145 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667: acquiring CA information for follower catalog manager: success
I20260927 12:41:13.211860 7145 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 12:41:13.291272 7061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:41:13.291579 7061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:41:13.291647 7061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:41:13.295733 7061 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:41:13.295814 7061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:41:13.295848 7061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:41:13.295904 7061 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:41:13.295956 7061 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:41:13.301800 7061 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46303,127.0.98.61:46395,127.0.98.60:42401
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:42401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7061
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:41:13.302871 7061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:41:13.303903 7061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:41:13.312773 7151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:13.312770 7150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:13.312976 7061 server_base.cc:1034] running on GCE node
W20260927 12:41:13.313105 7153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:13.313709 7061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:41:13.314836 7061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:41:13.316027 7061 hybrid_clock.cc:648] HybridClock initialized: now 1790512873315991 us; error 53 us; skew 500 ppm
Sep 27 12:41:13 dist-test-slave-bnc1 krb5kdc[6450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512873, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:41:13.318737 7061 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260927 12:41:13.319569 7061 webserver.cc:460] Webserver started at http://127.0.98.60:40091/ using document root <none> and password file <none>
I20260927 12:41:13.320112 7061 fs_manager.cc:359] Metadata directory not provided
I20260927 12:41:13.320200 7061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:41:13.320468 7061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:41:13.322583 7061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/master-2/data/instance:
uuid: "aaa07f27c66948e89a3b73053bcaafc0"
format_stamp: "Formatted at 2026-09-27 12:41:13 on dist-test-slave-bnc1"
server_key: "b27742ec42d6739b69acbb6bddd8441c"
server_key_iv: "3ab2b0cf7a8875b15d259834b0465145"
server_key_version: "encryptionkey@0"
I20260927 12:41:13.323074 7061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/master-2/wal/instance:
uuid: "aaa07f27c66948e89a3b73053bcaafc0"
format_stamp: "Formatted at 2026-09-27 12:41:13 on dist-test-slave-bnc1"
server_key: "b27742ec42d6739b69acbb6bddd8441c"
server_key_iv: "3ab2b0cf7a8875b15d259834b0465145"
server_key_version: "encryptionkey@0"
I20260927 12:41:13.326660 7061 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260927 12:41:13.328892 7160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:13.330247 7061 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:41:13.330374 7061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/master-2/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/master-2/wal
uuid: "aaa07f27c66948e89a3b73053bcaafc0"
format_stamp: "Formatted at 2026-09-27 12:41:13 on dist-test-slave-bnc1"
server_key: "b27742ec42d6739b69acbb6bddd8441c"
server_key_iv: "3ab2b0cf7a8875b15d259834b0465145"
server_key_version: "encryptionkey@0"
I20260927 12:41:13.330482 7061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:41:13.359427 7061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:41:13.362598 7061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:41:13.362802 7061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:41:13.370244 7061 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:42401
I20260927 12:41:13.370576 7212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:42401 every 8 connection(s)
I20260927 12:41:13.371276 7061 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/master-2/data/info.pb
I20260927 12:41:13.374504 7213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:41:13.378791 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 7061
I20260927 12:41:13.378757 7213 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46303 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 12:41:13 dist-test-slave-bnc1 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512873, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 27 12:41:13 dist-test-slave-bnc1 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 12:41:13.391830 6534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:13.381711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44604 (local address 127.0.98.62:46303)
0927 12:41:13.381840 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:13.381845 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:13.382057 (+ 212us) server_negotiation.cc:407] Connection header received
0927 12:41:13.382264 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:13.382268 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:13.382320 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:13.382428 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:13.383215 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.383758 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.384781 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.385025 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.385142 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:13.387958 (+ 2816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:13.387969 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:13.387972 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:41:13.387997 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:13.389681 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:13.390327 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:13.390330 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:13.390331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:13.390399 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:13.390770 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:13.390775 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:13.390777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:13.390955 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:13.391061 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:13.391544 (+ 483us) server_negotiation.cc:299] Negotiation successful
0927 12:41:13.391658 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 12:41:13.392745 7216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:13.381526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46303 (local address 127.0.0.1:44604)
0927 12:41:13.381929 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:13.381967 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:13.382119 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:13.382475 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:13.382481 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:13.382821 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:13.383071 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:13.383084 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.383894 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.383897 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:13.384653 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:13.384662 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.385176 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.385180 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:13.385322 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:13.386246 (+ 924us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:13.386261 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:13.387793 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:41:13.389784 (+ 1991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:13.389793 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:13.389805 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:13.390229 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:13.390528 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:13.390531 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:13.390533 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:13.390675 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:13.391069 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:41:13.391079 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:41:13.391319 (+ 240us) client_negotiation.cc:769] Sending connection context
0927 12:41:13.391486 (+ 167us) client_negotiation.cc:240] Negotiation successful
0927 12:41:13.391792 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":73,"thread_start_us":136,"threads_started":1}
I20260927 12:41:13.392747 7215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:13.380288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39292 (local address 127.0.98.60:42401)
0927 12:41:13.380674 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:13.380690 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:13.380735 (+ 45us) server_negotiation.cc:407] Connection header received
0927 12:41:13.381648 (+ 913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:13.381672 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:13.382078 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:13.382346 (+ 268us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:13.383151 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.384004 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.384835 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.385232 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.385386 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:13.388377 (+ 2991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:13.388384 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:13.388395 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:41:13.388414 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:13.390253 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:13.390725 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:13.390730 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:13.390732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:13.390813 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:13.391114 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:13.391118 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:13.391120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:13.391365 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:13.391574 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:13.391779 (+ 205us) server_negotiation.cc:299] Negotiation successful
0927 12:41:13.391929 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":259,"thread_start_us":109,"threads_started":1}
I20260927 12:41:13.393626 6686 heartbeater.cc:503] Master 127.0.98.62:46303 was elected leader, sending a full tablet report...
I20260927 12:41:13.394397 7213 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46395 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 12:41:13 dist-test-slave-bnc1 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512873, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 12:41:13.402369 7216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:13.394916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46395 (local address 127.0.0.1:55824)
0927 12:41:13.395074 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:13.395085 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:13.395143 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:13.395532 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:13.395534 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:13.395662 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:13.395747 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:13.395752 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.396593 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.396595 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:13.397112 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:13.397118 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.397535 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.397538 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:13.397662 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:13.398130 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:13.398141 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:13.399269 (+ 1128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:41:13.400900 (+ 1631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:13.400903 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:13.400905 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:13.401098 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:13.401376 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:13.401381 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:13.401383 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:13.401467 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:13.401863 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:41:13.401866 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:41:13.401968 (+ 102us) client_negotiation.cc:769] Sending connection context
0927 12:41:13.402074 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 12:41:13.402202 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105}
I20260927 12:41:13.402416 7217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:13.395005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55824 (local address 127.0.98.61:46395)
0927 12:41:13.395266 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:13.395270 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:13.395284 (+ 14us) server_negotiation.cc:407] Connection header received
0927 12:41:13.395324 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:13.395327 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:13.395372 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:13.395498 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:13.395849 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.396505 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.397205 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.397445 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.397522 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:13.399426 (+ 1904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:13.399430 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:13.399432 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:41:13.399453 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:13.400779 (+ 1326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:13.401198 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:13.401203 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:13.401205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:13.401282 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:13.401591 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:13.401594 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:13.401596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:13.401766 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:13.401873 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:13.402134 (+ 261us) server_negotiation.cc:299] Negotiation successful
0927 12:41:13.402252 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":111,"threads_started":1}
I20260927 12:41:13.403204 7213 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42401 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 12:41:13 dist-test-slave-bnc1 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512873, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 12:41:13.411056 7216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:13.403671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42401 (local address 127.0.0.1:39300)
0927 12:41:13.403804 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:13.403819 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:13.403919 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:13.404153 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:13.404156 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:13.404310 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:13.404366 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:13.404370 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.405099 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.405105 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:13.405664 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:13.405671 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.406189 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.406192 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:13.406248 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:13.406719 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:13.406730 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:13.408000 (+ 1270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:41:13.409627 (+ 1627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:13.409630 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:13.409632 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:13.409891 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:13.410140 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:13.410143 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:13.410144 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:13.410219 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:13.410558 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:41:13.410561 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:41:13.410677 (+ 116us) client_negotiation.cc:769] Sending connection context
0927 12:41:13.410777 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 12:41:13.410914 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 12:41:13.411056 7215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:13.403720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39300 (local address 127.0.98.60:42401)
0927 12:41:13.403804 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:13.403809 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:13.403883 (+ 74us) server_negotiation.cc:407] Connection header received
0927 12:41:13.404025 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:13.404028 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:13.404067 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:13.404146 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:13.404471 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.404976 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.405834 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.406085 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.406174 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:13.408121 (+ 1947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:13.408124 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:13.408126 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:41:13.408145 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:13.409450 (+ 1305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:13.409984 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:13.409987 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:13.409988 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:13.410053 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:13.410297 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:13.410299 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:13.410300 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:13.410445 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:13.410530 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:13.410819 (+ 289us) server_negotiation.cc:299] Negotiation successful
0927 12:41:13.410931 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":39}
I20260927 12:41:13.414386 7213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aaa07f27c66948e89a3b73053bcaafc0: Bootstrap starting.
I20260927 12:41:13.416571 7213 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aaa07f27c66948e89a3b73053bcaafc0: Neither blocks nor log segments found. Creating new log.
I20260927 12:41:13.417224 7213 log.cc:824] T 00000000000000000000000000000000 P aaa07f27c66948e89a3b73053bcaafc0: Log is configured to *not* fsync() on all Append() calls
I20260927 12:41:13.419018 7213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aaa07f27c66948e89a3b73053bcaafc0: No bootstrap required, opened a new log
I20260927 12:41:13.421931 7213 raft_consensus.cc:348] T 00000000000000000000000000000000 P aaa07f27c66948e89a3b73053bcaafc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42be682b019e44ed8f1a42e5cf28499f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46303 } } peers { permanent_uuid: "04babc8b97154b25ac1e359afb91c667" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46395 } } peers { permanent_uuid: "aaa07f27c66948e89a3b73053bcaafc0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42401 } }
I20260927 12:41:13.422122 7213 raft_consensus.cc:374] T 00000000000000000000000000000000 P aaa07f27c66948e89a3b73053bcaafc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:41:13.422155 7213 raft_consensus.cc:729] T 00000000000000000000000000000000 P aaa07f27c66948e89a3b73053bcaafc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaa07f27c66948e89a3b73053bcaafc0, State: Initialized, Role: FOLLOWER
I20260927 12:41:13.422515 7213 consensus_queue.cc:260] T 00000000000000000000000000000000 P aaa07f27c66948e89a3b73053bcaafc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42be682b019e44ed8f1a42e5cf28499f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46303 } } peers { permanent_uuid: "04babc8b97154b25ac1e359afb91c667" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46395 } } peers { permanent_uuid: "aaa07f27c66948e89a3b73053bcaafc0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42401 } }
I20260927 12:41:13.422904 7219 sys_catalog.cc:455] T 00000000000000000000000000000000 P aaa07f27c66948e89a3b73053bcaafc0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42be682b019e44ed8f1a42e5cf28499f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46303 } } peers { permanent_uuid: "04babc8b97154b25ac1e359afb91c667" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46395 } } peers { permanent_uuid: "aaa07f27c66948e89a3b73053bcaafc0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42401 } } }
I20260927 12:41:13.423034 7219 sys_catalog.cc:458] T 00000000000000000000000000000000 P aaa07f27c66948e89a3b73053bcaafc0 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 12:41:13.423616 7213 sys_catalog.cc:564] T 00000000000000000000000000000000 P aaa07f27c66948e89a3b73053bcaafc0 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 12:41:13.428189 7230 catalog_manager.cc:1558] T 00000000000000000000000000000000 P aaa07f27c66948e89a3b73053bcaafc0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 12:41:13.428269 7230 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 12:41:13.431890 7061 master_runner.cc:428] Detected that this master aaa07f27c66948e89a3b73053bcaafc0 is joining an existing cluster
I20260927 12:41:13.431973 7061 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:42401
I20260927 12:41:13.444257 7237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:13.434314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46303 (local address 127.0.0.1:44612)
0927 12:41:13.434653 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:13.434665 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:13.434782 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:13.435170 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:13.435173 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:13.435423 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:13.435569 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:13.435579 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.436730 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.436732 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:13.437690 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:13.437698 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.438418 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.438421 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:13.438557 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:13.439692 (+ 1135us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:13.439703 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:13.440071 (+ 368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:41:13.442573 (+ 2502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:13.442578 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:13.442580 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:13.442836 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:13.443128 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:13.443131 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:13.443133 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:13.443227 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:13.443671 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:41:13.443674 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:41:13.443776 (+ 102us) client_negotiation.cc:769] Sending connection context
0927 12:41:13.443922 (+ 146us) client_negotiation.cc:240] Negotiation successful
0927 12:41:13.444061 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":49,"thread_start_us":110,"threads_started":1}
I20260927 12:41:13.444283 6534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:13.434273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44612 (local address 127.0.98.62:46303)
0927 12:41:13.434423 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:13.434428 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:13.434728 (+ 300us) server_negotiation.cc:407] Connection header received
0927 12:41:13.434907 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:13.434913 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:13.434968 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:13.435041 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:13.435712 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.436608 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.437819 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.438152 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.438253 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:13.440231 (+ 1978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:13.440238 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:13.440243 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:41:13.440273 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:13.442442 (+ 2169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:13.442938 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:13.442941 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:13.442943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:13.443027 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:13.443332 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:13.443335 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:13.443337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:13.443569 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:13.443671 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:13.443986 (+ 315us) server_negotiation.cc:299] Negotiation successful
0927 12:41:13.444115 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 12:41:13.444402 7217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:13.434693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55836 (local address 127.0.98.61:46395)
0927 12:41:13.434817 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:13.434821 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:13.434834 (+ 13us) server_negotiation.cc:407] Connection header received
0927 12:41:13.434879 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:13.434884 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:13.434932 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:13.435106 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:13.436173 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.436977 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.439037 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.439393 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.439466 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:13.440935 (+ 1469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:13.440940 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:13.440945 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:41:13.440979 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:13.442670 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:13.443143 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:13.443147 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:13.443149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:13.443226 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:13.443500 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:13.443504 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:13.443506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:13.443721 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:13.443850 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:13.444126 (+ 276us) server_negotiation.cc:299] Negotiation successful
0927 12:41:13.444242 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 12:41:13.445129 7236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:13.434263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46395 (local address 127.0.0.1:55836)
0927 12:41:13.434561 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:13.434579 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:13.434674 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:13.435658 (+ 984us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:13.435661 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:13.435870 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:13.436027 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:13.436035 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.437157 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.437160 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:13.438659 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:13.438669 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.439792 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.439795 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:13.439889 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:13.440388 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:13.440402 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:13.440794 (+ 392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:41:13.442771 (+ 1977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:13.442774 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:13.442777 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:13.443026 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:13.443328 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:13.443331 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:13.443333 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:13.443406 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:13.443851 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:41:13.443855 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:41:13.443945 (+ 90us) client_negotiation.cc:769] Sending connection context
0927 12:41:13.444798 (+ 853us) client_negotiation.cc:240] Negotiation successful
0927 12:41:13.444946 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"mutex_wait_us":69,"thread_start_us":119,"threads_started":1}
I20260927 12:41:13.446547 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 6557
I20260927 12:41:13.454061 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:42501
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:46303
--webserver_port=42085
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:46303,127.0.98.61:46395
--tserver_master_addrs=127.0.98.62:46303,127.0.98.61:46395,127.0.98.60:42401 with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 12:41:13 dist-test-slave-bnc1 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512872, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260927 12:41:13.470984 7052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:13.449228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42401 (local address 127.0.0.1:39318)
0927 12:41:13.449469 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:13.449484 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:13.449586 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:13.450310 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:13.450317 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:13.450543 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:13.450676 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:13.450686 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.451741 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.451747 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:13.452740 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:13.452753 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.453337 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.453344 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:13.453475 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:13.454025 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:13.454039 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:13.455495 (+ 1456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:41:13.469037 (+ 13542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:13.469041 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:13.469044 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:13.469315 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:13.469744 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:13.469750 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:13.469756 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:13.469866 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:13.470463 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:41:13.470466 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:41:13.470559 (+ 93us) client_negotiation.cc:769] Sending connection context
0927 12:41:13.470663 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 12:41:13.470822 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152}
I20260927 12:41:13.470993 7215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:13.435456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39308 (local address 127.0.98.60:42401)
0927 12:41:13.435564 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:13.435567 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:13.435765 (+ 198us) server_negotiation.cc:407] Connection header received
0927 12:41:13.435907 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:13.435910 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:13.435942 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:13.436032 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:13.436662 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.437431 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.438668 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.438946 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.439013 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:13.467358 (+ 28345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:13.467365 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:13.467371 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:41:13.467451 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:13.469157 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:13.469624 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:13.469628 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:13.469629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:13.469697 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:13.469969 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:13.469972 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:13.469973 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:13.470320 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:13.470417 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:13.470698 (+ 281us) server_negotiation.cc:299] Negotiation successful
0927 12:41:13.470812 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":38,"spinlock_wait_cycles":3584}
I20260927 12:41:13.471516 7238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:13.435429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42401 (local address 127.0.0.1:39308)
0927 12:41:13.435693 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:13.435706 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:13.435813 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:13.436247 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:13.436250 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:13.436455 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:13.436549 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:13.436555 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.437549 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.437552 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:13.438559 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:13.438567 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.439131 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.439134 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:13.439238 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:13.466577 (+ 27339us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:13.466597 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:13.467162 (+ 565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:41:13.469260 (+ 2098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:13.469265 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:13.469270 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:13.469503 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:13.469789 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:13.469794 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:13.469799 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:13.469867 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:13.470453 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:41:13.470455 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:41:13.470536 (+ 81us) client_negotiation.cc:769] Sending connection context
0927 12:41:13.471207 (+ 671us) client_negotiation.cc:240] Negotiation successful
0927 12:41:13.471334 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":49,"thread_start_us":93,"threads_started":1}
I20260927 12:41:13.472214 7239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:13.449735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39318 (local address 127.0.98.60:42401)
0927 12:41:13.450038 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:13.450049 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:13.450062 (+ 13us) server_negotiation.cc:407] Connection header received
0927 12:41:13.450125 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:13.450128 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:13.450176 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:13.450265 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:13.450795 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.451611 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.452861 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.453217 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.453319 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:13.455667 (+ 2348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:13.455670 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:13.455673 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:41:13.455698 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:13.468873 (+ 13175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:13.469492 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:13.469496 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:13.469498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:13.469637 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:13.469971 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:13.469973 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:13.469975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:13.470320 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:13.470417 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:13.471958 (+ 1541us) server_negotiation.cc:299] Negotiation successful
0927 12:41:13.472063 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":222,"spinlock_wait_cycles":2816,"thread_start_us":127,"threads_started":1}
I20260927 12:41:13.473249 6495 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:42401: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "aaa07f27c66948e89a3b73053bcaafc0"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 42401
}
attrs {
promote: true
}
}
dest_uuid: "42be682b019e44ed8f1a42e5cf28499f"
cas_config_opid_index: 5
I20260927 12:41:13.473906 6495 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42be682b019e44ed8f1a42e5cf28499f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "42be682b019e44ed8f1a42e5cf28499f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46303 } } peers { permanent_uuid: "04babc8b97154b25ac1e359afb91c667" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46395 } attrs { promote: true } } peers { permanent_uuid: "aaa07f27c66948e89a3b73053bcaafc0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42401 } attrs { promote: true } }
I20260927 12:41:13.475061 7054 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 42be682b019e44ed8f1a42e5cf28499f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER aaa07f27c66948e89a3b73053bcaafc0 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "42be682b019e44ed8f1a42e5cf28499f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46303 } } peers { permanent_uuid: "04babc8b97154b25ac1e359afb91c667" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46395 } attrs { promote: true } } peers { permanent_uuid: "aaa07f27c66948e89a3b73053bcaafc0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42401 } attrs { promote: true } } }
I20260927 12:41:13.476323 7054 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:42401
I20260927 12:41:13.476493 7054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42be682b019e44ed8f1a42e5cf28499f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "42be682b019e44ed8f1a42e5cf28499f" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "42be682b019e44ed8f1a42e5cf28499f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46303 } } peers { permanent_uuid: "04babc8b97154b25ac1e359afb91c667" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46395 } attrs { promote: true } } peers { permanent_uuid: "aaa07f27c66948e89a3b73053bcaafc0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42401 } attrs { promote: true } } }
I20260927 12:41:13.476613 7054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42be682b019e44ed8f1a42e5cf28499f [sys.catalog]: This master's current role is: LEADER
I20260927 12:41:13.477233 7061 master.cc:537] Master@127.0.98.60:42401 shutting down...
W20260927 12:41:13.478847 6480 consensus_peers.cc:487] T 00000000000000000000000000000000 P 42be682b019e44ed8f1a42e5cf28499f -> Peer aaa07f27c66948e89a3b73053bcaafc0 (127.0.98.60:42401): Couldn't send request to peer aaa07f27c66948e89a3b73053bcaafc0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260927 12:41:13.482702 7061 raft_consensus.cc:2229] T 00000000000000000000000000000000 P aaa07f27c66948e89a3b73053bcaafc0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260927 12:41:13.482990 7061 raft_consensus.cc:2258] T 00000000000000000000000000000000 P aaa07f27c66948e89a3b73053bcaafc0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260927 12:41:13.483080 7061 tablet_replica.cc:330] T 00000000000000000000000000000000 P aaa07f27c66948e89a3b73053bcaafc0: stopping tablet replica
I20260927 12:41:13.485307 7217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:13.476796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55850 (local address 127.0.98.61:46395)
0927 12:41:13.476924 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:13.476931 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:13.480101 (+ 3170us) server_negotiation.cc:407] Connection header received
0927 12:41:13.480290 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:13.480293 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:13.480355 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:13.480425 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:41:13.481197 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.481928 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.483596 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.484223 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.484352 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:13.484508 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:13.484861 (+ 353us) server_negotiation.cc:299] Negotiation successful
0927 12:41:13.484966 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 12:41:13.485970 7052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:13.475190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46395 (local address 127.0.0.1:55850)
0927 12:41:13.480023 (+ 4833us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:13.480040 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:13.480175 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:13.480855 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:13.480869 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:13.480905 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:41:13.481066 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:13.481074 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.482212 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.482217 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:13.483446 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:13.483461 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.484604 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.484607 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:13.484692 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:13.484737 (+ 45us) client_negotiation.cc:769] Sending connection context
0927 12:41:13.485743 (+ 1006us) client_negotiation.cc:240] Negotiation successful
0927 12:41:13.485838 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4007}
I20260927 12:41:13.486725 7105 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667 [term 1 LEARNER]: Refusing update from remote peer 42be682b019e44ed8f1a42e5cf28499f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 12:41:13.487375 7055 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 42be682b019e44ed8f1a42e5cf28499f [LEADER]: Connected to new peer: Peer: permanent_uuid: "04babc8b97154b25ac1e359afb91c667" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46395 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 12:41:13.488945 7061 master.cc:552] Master@127.0.98.60:42401 shutdown complete.
I20260927 12:41:13.489094 7061 master_runner.cc:305] Clearing existing system tablet
I20260927 12:41:13.489377 7061 fs_manager.cc:359] Metadata directory not provided
I20260927 12:41:13.489496 7061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:41:13.489735 7105 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER aaa07f27c66948e89a3b73053bcaafc0 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "42be682b019e44ed8f1a42e5cf28499f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46303 } } peers { permanent_uuid: "04babc8b97154b25ac1e359afb91c667" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46395 } attrs { promote: true } } peers { permanent_uuid: "aaa07f27c66948e89a3b73053bcaafc0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42401 } attrs { promote: true } } }
I20260927 12:41:13.490911 7061 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 12:41:13.491638 7134 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42be682b019e44ed8f1a42e5cf28499f. Latest consensus state: current_term: 1 leader_uuid: "42be682b019e44ed8f1a42e5cf28499f" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "42be682b019e44ed8f1a42e5cf28499f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46303 } } peers { permanent_uuid: "04babc8b97154b25ac1e359afb91c667" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46395 } attrs { promote: true } } peers { permanent_uuid: "aaa07f27c66948e89a3b73053bcaafc0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42401 } attrs { promote: true } } }
I20260927 12:41:13.491765 7134 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667 [sys.catalog]: This master's current role is: LEARNER
I20260927 12:41:13.491888 7244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:13.491858 7134 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "42be682b019e44ed8f1a42e5cf28499f" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "42be682b019e44ed8f1a42e5cf28499f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46303 } } peers { permanent_uuid: "04babc8b97154b25ac1e359afb91c667" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 46395 } attrs { promote: true } } peers { permanent_uuid: "aaa07f27c66948e89a3b73053bcaafc0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42401 } attrs { promote: true } } }
I20260927 12:41:13.491952 7134 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667 [sys.catalog]: This master's current role is: LEARNER
I20260927 12:41:13.492238 7061 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:41:13.492353 7061 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 12:41:13.492385 7061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/master-2/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/master-2/wal
uuid: "aaa07f27c66948e89a3b73053bcaafc0"
format_stamp: "Formatted at 2026-09-27 12:41:13 on dist-test-slave-bnc1"
server_key: "b27742ec42d6739b69acbb6bddd8441c"
server_key_iv: "3ab2b0cf7a8875b15d259834b0465145"
server_key_version: "encryptionkey@0"
I20260927 12:41:13.493325 7055 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 42be682b019e44ed8f1a42e5cf28499f: attempting to promote NON_VOTER 04babc8b97154b25ac1e359afb91c667 to VOTER
I20260927 12:41:13.493746 7055 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42be682b019e44ed8f1a42e5cf28499f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "42be682b019e44ed8f1a42e5cf28499f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46303 } } peers { permanent_uuid: "04babc8b97154b25ac1e359afb91c667" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46395 } attrs { promote: false } } peers { permanent_uuid: "aaa07f27c66948e89a3b73053bcaafc0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42401 } attrs { promote: true } }
I20260927 12:41:13.494413 7061 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P aaa07f27c66948e89a3b73053bcaafc0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:41:13.495445 7105 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667 [term 1 LEARNER]: Refusing update from remote peer 42be682b019e44ed8f1a42e5cf28499f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260927 12:41:13.495803 7054 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 42be682b019e44ed8f1a42e5cf28499f [LEADER]: Connected to new peer: Peer: permanent_uuid: "04babc8b97154b25ac1e359afb91c667" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260927 12:41:13.496845 7061 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P aaa07f27c66948e89a3b73053bcaafc0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 12:41:13.496917 7061 log.cc:1196] T 00000000000000000000000000000000 P aaa07f27c66948e89a3b73053bcaafc0: Deleting WAL directory at /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260927 12:41:13.496965 7052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:13.494747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42401 (local address 127.0.0.1:39320)
0927 12:41:13.496816 (+ 2069us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:13.496874 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1983}
I20260927 12:41:13.497167 7061 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P aaa07f27c66948e89a3b73053bcaafc0: Deleting consensus metadata
I20260927 12:41:13.497321 7061 master_runner.cc:315] Copying system tablet from 127.0.98.62:46303
I20260927 12:41:13.497844 7052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:13.497425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42401 (local address 127.0.0.1:39324)
0927 12:41:13.497732 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:13.497777 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":96,"threads_started":1}
W20260927 12:41:13.497942 6480 consensus_peers.cc:487] T 00000000000000000000000000000000 P 42be682b019e44ed8f1a42e5cf28499f -> Peer aaa07f27c66948e89a3b73053bcaafc0 (127.0.98.60:42401): Couldn't send request to peer aaa07f27c66948e89a3b73053bcaafc0. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42401: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 12:41:13.498950 7061 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P aaa07f27c66948e89a3b73053bcaafc0: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:46303
I20260927 12:41:13.498915 7054 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 42be682b019e44ed8f1a42e5cf28499f [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 04babc8b97154b25ac1e359afb91c667 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "42be682b019e44ed8f1a42e5cf28499f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46303 } } peers { permanent_uuid: "04babc8b97154b25ac1e359afb91c667" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46395 } attrs { promote: false } } peers { permanent_uuid: "aaa07f27c66948e89a3b73053bcaafc0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42401 } attrs { promote: true } } }
I20260927 12:41:13.500425 7054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42be682b019e44ed8f1a42e5cf28499f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "42be682b019e44ed8f1a42e5cf28499f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "42be682b019e44ed8f1a42e5cf28499f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46303 } } peers { permanent_uuid: "04babc8b97154b25ac1e359afb91c667" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46395 } attrs { promote: false } } peers { permanent_uuid: "aaa07f27c66948e89a3b73053bcaafc0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42401 } attrs { promote: true } } }
I20260927 12:41:13.500553 7054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42be682b019e44ed8f1a42e5cf28499f [sys.catalog]: This master's current role is: LEADER
I20260927 12:41:13.500703 7054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42be682b019e44ed8f1a42e5cf28499f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "42be682b019e44ed8f1a42e5cf28499f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "42be682b019e44ed8f1a42e5cf28499f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46303 } } peers { permanent_uuid: "04babc8b97154b25ac1e359afb91c667" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46395 } attrs { promote: false } } peers { permanent_uuid: "aaa07f27c66948e89a3b73053bcaafc0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42401 } attrs { promote: true } } }
I20260927 12:41:13.500792 7054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42be682b019e44ed8f1a42e5cf28499f [sys.catalog]: This master's current role is: LEADER
I20260927 12:41:13.501451 7105 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 04babc8b97154b25ac1e359afb91c667 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "42be682b019e44ed8f1a42e5cf28499f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46303 } } peers { permanent_uuid: "04babc8b97154b25ac1e359afb91c667" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46395 } attrs { promote: false } } peers { permanent_uuid: "aaa07f27c66948e89a3b73053bcaafc0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42401 } attrs { promote: true } } }
I20260927 12:41:13.503134 7243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "42be682b019e44ed8f1a42e5cf28499f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "42be682b019e44ed8f1a42e5cf28499f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46303 } } peers { permanent_uuid: "04babc8b97154b25ac1e359afb91c667" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46395 } attrs { promote: false } } peers { permanent_uuid: "aaa07f27c66948e89a3b73053bcaafc0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42401 } attrs { promote: true } } }
I20260927 12:41:13.503247 7243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 12:41:13.508347 6534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:13.499751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44620 (local address 127.0.98.62:46303)
0927 12:41:13.499847 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:13.499851 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:13.500399 (+ 548us) server_negotiation.cc:407] Connection header received
0927 12:41:13.500692 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:13.500696 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:13.500748 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:13.500804 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:13.501540 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.502444 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.503641 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.503875 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.504000 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:13.505151 (+ 1151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:13.505154 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:13.505157 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:41:13.505181 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:13.506731 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:13.507135 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:13.507140 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:13.507142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:13.507226 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:13.507521 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:13.507524 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:13.507526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:13.507699 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:13.507818 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:13.508094 (+ 276us) server_negotiation.cc:299] Negotiation successful
0927 12:41:13.508197 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 12:41:13.508369 7252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:13.499948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46303 (local address 127.0.0.1:44620)
0927 12:41:13.500303 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:13.500317 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:13.500478 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:13.501019 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:13.501024 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:13.501248 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:13.501390 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:13.501412 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.502596 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.502602 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:13.503535 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:13.503546 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.504000 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.504004 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:13.504095 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:13.504644 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:13.504664 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:13.504982 (+ 318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:41:13.506821 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:13.506826 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:13.506828 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:13.507035 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:13.507331 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:13.507334 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:13.507336 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:13.507420 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:13.507836 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:41:13.507839 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:41:13.507934 (+ 95us) client_negotiation.cc:769] Sending connection context
0927 12:41:13.508042 (+ 108us) client_negotiation.cc:240] Negotiation successful
0927 12:41:13.508184 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":64,"threads_started":1}
I20260927 12:41:13.509112 6515 tablet_copy_service.cc:140] P 42be682b019e44ed8f1a42e5cf28499f: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer aaa07f27c66948e89a3b73053bcaafc0 ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:44620)
I20260927 12:41:13.509222 6515 tablet_copy_service.cc:161] P 42be682b019e44ed8f1a42e5cf28499f: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer aaa07f27c66948e89a3b73053bcaafc0 at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:44620: session id = aaa07f27c66948e89a3b73053bcaafc0-00000000000000000000000000000000
I20260927 12:41:13.510247 6515 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 42be682b019e44ed8f1a42e5cf28499f: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 12:41:13.511451 7061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 12:41:13.513332 7061 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P aaa07f27c66948e89a3b73053bcaafc0: tablet copy: Starting download of 0 data blocks...
I20260927 12:41:13.513558 7061 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P aaa07f27c66948e89a3b73053bcaafc0: tablet copy: Starting download of 1 WAL segments...
I20260927 12:41:13.515012 7061 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P aaa07f27c66948e89a3b73053bcaafc0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 12:41:13.516531 6515 tablet_copy_service.cc:342] P 42be682b019e44ed8f1a42e5cf28499f: Request end of tablet copy session aaa07f27c66948e89a3b73053bcaafc0-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:44620
I20260927 12:41:13.516629 6515 tablet_copy_service.cc:433] P 42be682b019e44ed8f1a42e5cf28499f: ending tablet copy session aaa07f27c66948e89a3b73053bcaafc0-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer aaa07f27c66948e89a3b73053bcaafc0
I20260927 12:41:13.523156 7061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:41:13.524784 7253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:13.524832 7256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:13.525373 7061 server_base.cc:1034] running on GCE node
W20260927 12:41:13.530053 7254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:13.530277 7061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:41:13.530656 7061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:41:13.531800 7061 hybrid_clock.cc:648] HybridClock initialized: now 1790512873531793 us; error 24 us; skew 500 ppm
Sep 27 12:41:13 dist-test-slave-bnc1 krb5kdc[6450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512873, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:41:13.533497 7061 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260927 12:41:13.534044 7061 webserver.cc:460] Webserver started at http://127.0.98.60:38977/ using document root <none> and password file <none>
I20260927 12:41:13.534266 7061 fs_manager.cc:359] Metadata directory not provided
I20260927 12:41:13.534324 7061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:41:13.535581 7061 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260927 12:41:13.536614 7263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:13.536863 7061 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 12:41:13.536962 7061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/master-2/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/master-2/wal
uuid: "aaa07f27c66948e89a3b73053bcaafc0"
format_stamp: "Formatted at 2026-09-27 12:41:13 on dist-test-slave-bnc1"
server_key: "b27742ec42d6739b69acbb6bddd8441c"
server_key_iv: "3ab2b0cf7a8875b15d259834b0465145"
server_key_version: "encryptionkey@0"
I20260927 12:41:13.537075 7061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:41:13.544852 7061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:41:13.547112 7061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:41:13.552376 7061 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:42401
I20260927 12:41:13.552385 7315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:42401 every 8 connection(s)
I20260927 12:41:13.553004 7061 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/master-2/data/info.pb
I20260927 12:41:13.554607 7316 sys_catalog.cc:263] Verifying existing consensus state
I20260927 12:41:13.555676 7316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aaa07f27c66948e89a3b73053bcaafc0: Bootstrap starting.
I20260927 12:41:13.565299 7316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aaa07f27c66948e89a3b73053bcaafc0: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 12:41:13.565704 7316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aaa07f27c66948e89a3b73053bcaafc0: Bootstrap complete.
I20260927 12:41:13.566269 7316 raft_consensus.cc:348] T 00000000000000000000000000000000 P aaa07f27c66948e89a3b73053bcaafc0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "42be682b019e44ed8f1a42e5cf28499f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46303 } } peers { permanent_uuid: "04babc8b97154b25ac1e359afb91c667" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46395 } attrs { promote: false } } peers { permanent_uuid: "aaa07f27c66948e89a3b73053bcaafc0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42401 } attrs { promote: true } }
I20260927 12:41:13.566431 7316 raft_consensus.cc:729] T 00000000000000000000000000000000 P aaa07f27c66948e89a3b73053bcaafc0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: aaa07f27c66948e89a3b73053bcaafc0, State: Initialized, Role: LEARNER
I20260927 12:41:13.566560 7316 consensus_queue.cc:260] T 00000000000000000000000000000000 P aaa07f27c66948e89a3b73053bcaafc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "42be682b019e44ed8f1a42e5cf28499f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46303 } } peers { permanent_uuid: "04babc8b97154b25ac1e359afb91c667" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46395 } attrs { promote: false } } peers { permanent_uuid: "aaa07f27c66948e89a3b73053bcaafc0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42401 } attrs { promote: true } }
I20260927 12:41:13.566918 7319 sys_catalog.cc:455] T 00000000000000000000000000000000 P aaa07f27c66948e89a3b73053bcaafc0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "42be682b019e44ed8f1a42e5cf28499f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46303 } } peers { permanent_uuid: "04babc8b97154b25ac1e359afb91c667" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46395 } attrs { promote: false } } peers { permanent_uuid: "aaa07f27c66948e89a3b73053bcaafc0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42401 } attrs { promote: true } } }
I20260927 12:41:13.567111 7316 sys_catalog.cc:564] T 00000000000000000000000000000000 P aaa07f27c66948e89a3b73053bcaafc0 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:41:13.567124 7319 sys_catalog.cc:458] T 00000000000000000000000000000000 P aaa07f27c66948e89a3b73053bcaafc0 [sys.catalog]: This master's current role is: LEARNER
I20260927 12:41:13.570751 7330 catalog_manager.cc:1259] Loaded cluster ID: d7aa23e71b254e6d8a29824dd7eff1f3
I20260927 12:41:13.570891 7330 catalog_manager.cc:1552] T 00000000000000000000000000000000 P aaa07f27c66948e89a3b73053bcaafc0: loading cluster ID for follower catalog manager: success
I20260927 12:41:13.572654 7330 catalog_manager.cc:1574] T 00000000000000000000000000000000 P aaa07f27c66948e89a3b73053bcaafc0: acquiring CA information for follower catalog manager: success
I20260927 12:41:13.573910 7330 catalog_manager.cc:1602] T 00000000000000000000000000000000 P aaa07f27c66948e89a3b73053bcaafc0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 12:41:13.602154 6820 heartbeater.cc:503] Master 127.0.98.62:46303 was elected leader, sending a full tablet report...
W20260927 12:41:13.608896 7240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:41:13.609256 7240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:41:13.609318 7240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:41:13.615530 7240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:41:13.615622 7240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:41:13.615712 7240 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:41:13.624483 7240 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:42501
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=42085
--tserver_master_addrs=127.0.98.62:46303,127.0.98.61:46395,127.0.98.60:42401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7240
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:41:13.625936 7240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:41:13.627099 7240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:41:13.636921 7335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:13.636929 7338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:13.636981 7336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:13.637490 7240 server_base.cc:1034] running on GCE node
I20260927 12:41:13.637938 7240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:41:13.638692 7240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:41:13.639853 7240 hybrid_clock.cc:648] HybridClock initialized: now 1790512873639840 us; error 28 us; skew 500 ppm
Sep 27 12:41:13 dist-test-slave-bnc1 krb5kdc[6450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512873, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:41:13.642743 7240 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 12:41:13.643589 7240 webserver.cc:460] Webserver started at http://127.0.98.1:42085/ using document root <none> and password file <none>
I20260927 12:41:13.644155 7240 fs_manager.cc:359] Metadata directory not provided
I20260927 12:41:13.644258 7240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:41:13.648054 7240 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:41:13.650127 7345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:13.651410 7240 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:41:13.651568 7240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-0/wal
uuid: "4ac28926eacd419bad1e9f8b5e063a36"
format_stamp: "Formatted at 2026-09-27 12:41:12 on dist-test-slave-bnc1"
server_key: "9db758d680f597c5fd5d7cbc6f7ded87"
server_key_iv: "25cbd11a48ae3e6062cf931fb26975f0"
server_key_version: "encryptionkey@0"
I20260927 12:41:13.652091 7240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:41:13.662681 7240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:41:13.665848 7240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:41:13.666038 7240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:41:13.666571 7240 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:41:13.667531 7240 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:41:13.667598 7240 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:13.667685 7240 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:41:13.667727 7240 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:13.677371 7240 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42501
I20260927 12:41:13.677690 7458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42501 every 8 connection(s)
I20260927 12:41:13.678491 7240 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:41:13.683418 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 7240
I20260927 12:41:13.683627 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 6691
Sep 27 12:41:13 dist-test-slave-bnc1 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512873, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 12:41:13.691272 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:43043
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:46303
--webserver_port=33365
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:46303,127.0.98.61:46395
--tserver_master_addrs=127.0.98.62:46303,127.0.98.61:46395,127.0.98.60:42401 with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 12:41:13 dist-test-slave-bnc1 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512873, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 27 12:41:13 dist-test-slave-bnc1 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512873, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 12:41:13.696959 6534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:13.680715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49265 (local address 127.0.98.62:46303)
0927 12:41:13.680842 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:13.680847 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:13.682038 (+ 1191us) server_negotiation.cc:407] Connection header received
0927 12:41:13.683361 (+ 1323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:13.683365 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:13.683420 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:13.683504 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:13.686580 (+ 3076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.687410 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.688479 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.688793 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.688874 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:13.693207 (+ 4333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:13.693212 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:13.693217 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:41:13.693252 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:13.695079 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:13.695657 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:13.695660 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:13.695662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:13.695738 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:13.696035 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:13.696039 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:13.696040 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:13.696235 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:13.696318 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:13.696660 (+ 342us) server_negotiation.cc:299] Negotiation successful
0927 12:41:13.696788 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 12:41:13.697353 7463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:13.680417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38609 (local address 127.0.98.60:42401)
0927 12:41:13.681004 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:13.681009 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:13.683418 (+ 2409us) server_negotiation.cc:407] Connection header received
0927 12:41:13.683463 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:13.683466 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:13.683513 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:13.683597 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:13.685543 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.686700 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.687913 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.688266 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.688358 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:13.692243 (+ 3885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:13.692247 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:13.692251 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:41:13.692283 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:13.694200 (+ 1917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:13.694978 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:13.694984 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:13.694998 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:13.695088 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:13.695530 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:13.695534 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:13.695536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:13.695783 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:13.695873 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:13.696775 (+ 902us) server_negotiation.cc:299] Negotiation successful
0927 12:41:13.696891 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":87,"threads_started":1}
I20260927 12:41:13.697806 7217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:13.681076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33655 (local address 127.0.98.61:46395)
0927 12:41:13.681206 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:13.681213 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:13.682077 (+ 864us) server_negotiation.cc:407] Connection header received
0927 12:41:13.683046 (+ 969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:13.683048 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:13.683093 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:13.683219 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:13.685357 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.686151 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.687917 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.688245 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.688321 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:13.692308 (+ 3987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:13.692315 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:13.692321 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:41:13.692355 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:13.694345 (+ 1990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:13.694981 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:13.694984 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:13.694988 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:13.695073 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:13.695533 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:13.695539 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:13.695544 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:13.695887 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:13.695984 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:13.697388 (+ 1404us) server_negotiation.cc:299] Negotiation successful
0927 12:41:13.697541 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 12:41:13.698138 7464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:13.680902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46395 (local address 127.0.98.1:33655)
0927 12:41:13.681871 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:13.681926 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:13.682813 (+ 887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:13.683418 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:13.683428 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:13.684461 (+ 1033us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:13.685182 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:13.685207 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.686330 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.686334 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:13.687784 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:13.687795 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.688510 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.688514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:13.688771 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:13.689537 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:13.689557 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:13.692123 (+ 2566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:41:13.694479 (+ 2356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:13.694483 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:13.694485 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:13.694851 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:13.695182 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:13.695188 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:13.695192 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:13.695419 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:13.696030 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:41:13.696036 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:41:13.696285 (+ 249us) client_negotiation.cc:769] Sending connection context
0927 12:41:13.696554 (+ 269us) client_negotiation.cc:240] Negotiation successful
0927 12:41:13.696899 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":878,"mutex_wait_us":232,"spinlock_wait_cycles":34816,"thread_start_us":122,"threads_started":1}
I20260927 12:41:13.698607 7466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:13.680902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46303 (local address 127.0.98.1:49265)
0927 12:41:13.681871 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:13.681926 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:13.682816 (+ 890us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:13.686095 (+ 3279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:13.686098 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:13.686275 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:13.686448 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:13.686454 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.687542 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.687545 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:13.688355 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:13.688363 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.689044 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.689050 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:13.689153 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:13.689884 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:13.689897 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:13.692998 (+ 3101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:41:13.695182 (+ 2184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:13.695187 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:13.695192 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:13.695542 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:13.695850 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:13.695852 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:13.695854 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:13.695933 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:13.696373 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:41:13.696375 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:41:13.696472 (+ 97us) client_negotiation.cc:769] Sending connection context
0927 12:41:13.697125 (+ 653us) client_negotiation.cc:240] Negotiation successful
0927 12:41:13.697261 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":761,"mutex_wait_us":5,"spinlock_wait_cycles":20480,"thread_start_us":134,"threads_started":1}
I20260927 12:41:13.698652 7465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:13.680902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42401 (local address 127.0.98.1:38609)
0927 12:41:13.683242 (+ 2340us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:13.683255 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:13.683345 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:13.684130 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:13.684133 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:13.685318 (+ 1185us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:13.685428 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:13.685434 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.686832 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.686835 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:13.687786 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:13.687795 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.688510 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.688513 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:13.688772 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:13.689447 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:13.689463 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:13.692043 (+ 2580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:41:13.694344 (+ 2301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:13.694351 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:13.694369 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:13.694849 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:13.695211 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:13.695213 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:13.695215 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:13.695410 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:13.695919 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:41:13.695926 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:41:13.696282 (+ 356us) client_negotiation.cc:769] Sending connection context
0927 12:41:13.696495 (+ 213us) client_negotiation.cc:240] Negotiation successful
0927 12:41:13.698145 (+ 1650us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2234,"mutex_wait_us":1548,"spinlock_wait_cycles":11776,"thread_start_us":86,"threads_started":1}
I20260927 12:41:13.699625 7461 heartbeater.cc:347] Connected to a master server at 127.0.98.61:46395
I20260927 12:41:13.699738 7459 heartbeater.cc:347] Connected to a master server at 127.0.98.60:42401
I20260927 12:41:13.699753 7460 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46303
I20260927 12:41:13.699873 7460 heartbeater.cc:464] Registering TS with master...
I20260927 12:41:13.699873 7459 heartbeater.cc:464] Registering TS with master...
I20260927 12:41:13.699873 7461 heartbeater.cc:464] Registering TS with master...
I20260927 12:41:13.700569 7459 heartbeater.cc:511] Master 127.0.98.60:42401 requested a full tablet report, sending...
I20260927 12:41:13.700572 7461 heartbeater.cc:511] Master 127.0.98.61:46395 requested a full tablet report, sending...
I20260927 12:41:13.700773 7460 heartbeater.cc:511] Master 127.0.98.62:46303 requested a full tablet report, sending...
I20260927 12:41:13.701871 6495 ts_manager.cc:194] Re-registered known tserver with Master: 4ac28926eacd419bad1e9f8b5e063a36 (127.0.98.1:42501)
I20260927 12:41:13.702334 7088 ts_manager.cc:194] Registered new tserver with Master: 4ac28926eacd419bad1e9f8b5e063a36 (127.0.98.1:42501)
I20260927 12:41:13.702417 6495 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49265
I20260927 12:41:13.703260 7276 ts_manager.cc:194] Registered new tserver with Master: 4ac28926eacd419bad1e9f8b5e063a36 (127.0.98.1:42501)
W20260927 12:41:13.806459 7467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:41:13.806725 7467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:41:13.806805 7467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:41:13.810918 7467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:41:13.810987 7467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:41:13.811066 7467 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:41:13.816172 7467 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:43043
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=33365
--tserver_master_addrs=127.0.98.62:46303,127.0.98.61:46395,127.0.98.60:42401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:41:13.817194 7467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:41:13.818197 7467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 12:41:13.825673 6954 heartbeater.cc:503] Master 127.0.98.62:46303 was elected leader, sending a full tablet report...
W20260927 12:41:13.827450 7475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:13.827481 7473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:13.827536 7472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:13.827793 7467 server_base.cc:1034] running on GCE node
I20260927 12:41:13.828146 7467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:41:13.828771 7467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:41:13.829936 7467 hybrid_clock.cc:648] HybridClock initialized: now 1790512873829914 us; error 32 us; skew 500 ppm
Sep 27 12:41:13 dist-test-slave-bnc1 krb5kdc[6450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512873, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:41:13.832533 7467 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 12:41:13.833420 7467 webserver.cc:460] Webserver started at http://127.0.98.2:33365/ using document root <none> and password file <none>
I20260927 12:41:13.834014 7467 fs_manager.cc:359] Metadata directory not provided
I20260927 12:41:13.834118 7467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:41:13.837922 7467 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260927 12:41:13.839902 7482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:13.841054 7467 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:41:13.841182 7467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-1/wal
uuid: "9fd3b55ca4154ec69ee88b69c88fb053"
format_stamp: "Formatted at 2026-09-27 12:41:12 on dist-test-slave-bnc1"
server_key: "69624e0f644ed3c6f23750ac607b790d"
server_key_iv: "994cdef9d764189eb8a54613c9c968a1"
server_key_version: "encryptionkey@0"
I20260927 12:41:13.841763 7467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:41:13.861778 7467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:41:13.865075 7467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:41:13.865302 7467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:41:13.865947 7467 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:41:13.866842 7467 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:41:13.866909 7467 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:13.866979 7467 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:41:13.867029 7467 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:13.876642 7467 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43043
I20260927 12:41:13.876941 7595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43043 every 8 connection(s)
I20260927 12:41:13.877854 7467 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:41:13.879731 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 7467
I20260927 12:41:13.879920 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 6825
I20260927 12:41:13.888586 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:40415
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:46303
--webserver_port=45313
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:46303,127.0.98.61:46395
--tserver_master_addrs=127.0.98.62:46303,127.0.98.61:46395,127.0.98.60:42401 with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 12:41:13 dist-test-slave-bnc1 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512873, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 27 12:41:13 dist-test-slave-bnc1 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512873, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 27 12:41:13 dist-test-slave-bnc1 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512873, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 12:41:13.897012 7217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:13.881156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59711 (local address 127.0.98.61:46395)
0927 12:41:13.882235 (+ 1079us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:13.882240 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:13.882254 (+ 14us) server_negotiation.cc:407] Connection header received
0927 12:41:13.882300 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:13.882306 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:13.882361 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:13.882443 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:13.884345 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.885163 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.887163 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.887499 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.887573 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:13.891737 (+ 4164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:13.891741 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:13.891744 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:41:13.891775 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:13.894388 (+ 2613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:13.895036 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:13.895040 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:13.895042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:13.895130 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:13.895546 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:13.895550 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:13.895552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:13.895799 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:13.895886 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:13.896694 (+ 808us) server_negotiation.cc:299] Negotiation successful
0927 12:41:13.896822 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":995}
I20260927 12:41:13.897445 6534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:13.881232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56019 (local address 127.0.98.62:46303)
0927 12:41:13.881372 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:13.881379 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:13.883835 (+ 2456us) server_negotiation.cc:407] Connection header received
0927 12:41:13.884029 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:13.884033 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:13.884084 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:13.884137 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:13.884724 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.885466 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.887172 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.887499 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.887573 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:13.893045 (+ 5472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:13.893049 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:13.893053 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:41:13.893098 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:13.894797 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:13.895278 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:13.895284 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:13.895289 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:13.895364 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:13.895662 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:13.895665 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:13.895666 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:13.895855 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:13.895932 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:13.896994 (+ 1062us) server_negotiation.cc:299] Negotiation successful
0927 12:41:13.897200 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 12:41:13.897836 7463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:13.879593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36125 (local address 127.0.98.60:42401)
0927 12:41:13.879755 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:13.879761 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:13.882192 (+ 2431us) server_negotiation.cc:407] Connection header received
0927 12:41:13.882398 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:13.882404 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:13.882475 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:13.882560 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:13.884363 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.885336 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.887184 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.887621 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.887721 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:13.892084 (+ 4363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:13.892090 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:13.892094 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:41:13.892130 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:13.895465 (+ 3335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:13.896043 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:13.896049 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:13.896052 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:13.896144 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:13.896481 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:13.896485 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:13.896487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:13.896749 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:13.896845 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:13.897501 (+ 656us) server_negotiation.cc:299] Negotiation successful
0927 12:41:13.897679 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 12:41:13.899039 7602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:13.880128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46303 (local address 127.0.98.2:56019)
0927 12:41:13.883741 (+ 3613us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:13.883756 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:13.883902 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:13.884286 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:13.884291 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:13.884491 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:13.884591 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:13.884597 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.885620 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.885623 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:13.887023 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:13.887033 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.887779 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.887783 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:13.889954 (+ 2171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:13.890499 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:13.890521 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:13.892895 (+ 2374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:41:13.894921 (+ 2026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:13.894924 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:13.894926 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:13.895168 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:13.895469 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:13.895475 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:13.895480 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:13.895559 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:13.895978 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:41:13.895980 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:41:13.896370 (+ 390us) client_negotiation.cc:769] Sending connection context
0927 12:41:13.897696 (+ 1326us) client_negotiation.cc:240] Negotiation successful
0927 12:41:13.897880 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3521,"mutex_wait_us":1532,"spinlock_wait_cycles":8320,"thread_start_us":90,"threads_started":1}
I20260927 12:41:13.899966 7600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:13.880128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46395 (local address 127.0.98.2:59711)
0927 12:41:13.880705 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:13.880756 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:13.881947 (+ 1191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:13.882633 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:13.882645 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:13.883249 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:13.884182 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:13.884203 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.885344 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.885347 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:13.887026 (+ 1679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:13.887033 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.887779 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.887783 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:13.888029 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:13.888875 (+ 846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:13.888902 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:13.891517 (+ 2615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:41:13.894523 (+ 3006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:13.894531 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:13.894551 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:13.894914 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:13.895245 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:13.895251 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:13.895256 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:13.895440 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:13.895957 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:41:13.895965 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:41:13.896370 (+ 405us) client_negotiation.cc:769] Sending connection context
0927 12:41:13.897184 (+ 814us) client_negotiation.cc:240] Negotiation successful
0927 12:41:13.899513 (+ 2329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"mutex_wait_us":565,"spinlock_wait_cycles":21248,"thread_start_us":121,"threads_started":1}
I20260927 12:41:13.900382 7597 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46303
I20260927 12:41:13.900529 7598 heartbeater.cc:347] Connected to a master server at 127.0.98.61:46395
I20260927 12:41:13.900632 7597 heartbeater.cc:464] Registering TS with master...
I20260927 12:41:13.900631 7598 heartbeater.cc:464] Registering TS with master...
I20260927 12:41:13.901243 7598 heartbeater.cc:511] Master 127.0.98.61:46395 requested a full tablet report, sending...
I20260927 12:41:13.901243 7597 heartbeater.cc:511] Master 127.0.98.62:46303 requested a full tablet report, sending...
I20260927 12:41:13.902426 6495 ts_manager.cc:194] Re-registered known tserver with Master: 9fd3b55ca4154ec69ee88b69c88fb053 (127.0.98.2:43043)
I20260927 12:41:13.902985 6495 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56019
I20260927 12:41:13.903541 7088 ts_manager.cc:194] Registered new tserver with Master: 9fd3b55ca4154ec69ee88b69c88fb053 (127.0.98.2:43043)
I20260927 12:41:13.904249 7246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:13.893231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42401 (local address 127.0.0.1:39338)
0927 12:41:13.893501 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:13.893514 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:13.893670 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:13.898408 (+ 4738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:13.898411 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:13.898431 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:41:13.898533 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:13.898542 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.900831 (+ 2289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.900834 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:13.902245 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:13.902261 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.903719 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.903724 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:13.903863 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:13.903902 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 12:41:13.903978 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 12:41:13.904078 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185}
I20260927 12:41:13.904914 7463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:13.893832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39338 (local address 127.0.98.60:42401)
0927 12:41:13.898003 (+ 4171us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:13.898011 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:13.898025 (+ 14us) server_negotiation.cc:407] Connection header received
0927 12:41:13.898073 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:13.898076 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:13.898139 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:13.898237 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:41:13.899048 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.899813 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.902459 (+ 2646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.903376 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.903503 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:13.904590 (+ 1087us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:13.904634 (+ 44us) server_negotiation.cc:299] Negotiation successful
0927 12:41:13.904738 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4098,"thread_start_us":94,"threads_started":1}
I20260927 12:41:13.905419 7601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:13.881331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42401 (local address 127.0.98.2:36125)
0927 12:41:13.881668 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:13.881684 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:13.882271 (+ 587us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:13.882698 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:13.882704 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:13.883266 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:13.884182 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:13.884205 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.885490 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.885496 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:13.887034 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:13.887047 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:13.887910 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:13.887916 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:13.888141 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:13.888909 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:13.888921 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:13.891917 (+ 2996us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:41:13.895609 (+ 3692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:13.895614 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:13.895617 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:13.895915 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:13.896268 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:13.896273 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:13.896275 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:13.896366 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:13.896906 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:41:13.896910 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:41:13.897000 (+ 90us) client_negotiation.cc:769] Sending connection context
0927 12:41:13.905050 (+ 8050us) client_negotiation.cc:240] Negotiation successful
0927 12:41:13.905247 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"mutex_wait_us":228,"spinlock_wait_cycles":10752,"thread_start_us":126,"threads_started":1}
I20260927 12:41:13.905670 7288 raft_consensus.cc:1203] T 00000000000000000000000000000000 P aaa07f27c66948e89a3b73053bcaafc0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260927 12:41:13.906152 7319 sys_catalog.cc:455] T 00000000000000000000000000000000 P aaa07f27c66948e89a3b73053bcaafc0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42be682b019e44ed8f1a42e5cf28499f. Latest consensus state: current_term: 1 leader_uuid: "42be682b019e44ed8f1a42e5cf28499f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "42be682b019e44ed8f1a42e5cf28499f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46303 } } peers { permanent_uuid: "04babc8b97154b25ac1e359afb91c667" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46395 } attrs { promote: false } } peers { permanent_uuid: "aaa07f27c66948e89a3b73053bcaafc0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42401 } attrs { promote: true } } }
I20260927 12:41:13.906289 7319 sys_catalog.cc:458] T 00000000000000000000000000000000 P aaa07f27c66948e89a3b73053bcaafc0 [sys.catalog]: This master's current role is: LEARNER
I20260927 12:41:13.906282 7596 heartbeater.cc:347] Connected to a master server at 127.0.98.60:42401
I20260927 12:41:13.906358 7596 heartbeater.cc:464] Registering TS with master...
I20260927 12:41:13.906529 7596 heartbeater.cc:511] Master 127.0.98.60:42401 requested a full tablet report, sending...
I20260927 12:41:13.907213 7276 ts_manager.cc:194] Registered new tserver with Master: 9fd3b55ca4154ec69ee88b69c88fb053 (127.0.98.2:43043)
W20260927 12:41:14.004956 7603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:41:14.005322 7603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:41:14.005403 7603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:41:14.009625 7603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:41:14.009696 7603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:41:14.009801 7603 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:41:14.015439 7603 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:40415
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=45313
--tserver_master_addrs=127.0.98.62:46303,127.0.98.61:46395,127.0.98.60:42401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7603
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:41:14.016602 7603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:41:14.017691 7603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:41:14.027746 7610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:14.027712 7609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:14.027930 7603 server_base.cc:1034] running on GCE node
W20260927 12:41:14.027737 7612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:14.028414 7603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:41:14.029073 7603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:41:14.030236 7603 hybrid_clock.cc:648] HybridClock initialized: now 1790512874030216 us; error 33 us; skew 500 ppm
Sep 27 12:41:14 dist-test-slave-bnc1 krb5kdc[6450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512874, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:41:14.033268 7603 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 12:41:14.034291 7603 webserver.cc:460] Webserver started at http://127.0.98.3:45313/ using document root <none> and password file <none>
I20260927 12:41:14.034821 7603 fs_manager.cc:359] Metadata directory not provided
I20260927 12:41:14.034890 7603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:41:14.038762 7603 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 12:41:14.040635 7619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:14.041870 7603 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 12:41:14.041957 7603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-2/wal
uuid: "849bb514fb834ec38a1f428e351782d0"
format_stamp: "Formatted at 2026-09-27 12:41:12 on dist-test-slave-bnc1"
server_key: "4d0511ff1c623af551429d3b25c2084d"
server_key_iv: "06f2ad7ac3969b37998aeba6cc1ed363"
server_key_version: "encryptionkey@0"
I20260927 12:41:14.042728 7603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:41:14.062821 7603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:41:14.066395 7603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:41:14.066552 7603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:41:14.067173 7603 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:41:14.068102 7603 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:41:14.068148 7603 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:14.068189 7603 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:41:14.068213 7603 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:14.077185 7603 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40415
I20260927 12:41:14.077211 7732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40415 every 8 connection(s)
I20260927 12:41:14.078245 7603 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:41:14.085166 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 7603
Sep 27 12:41:14 dist-test-slave-bnc1 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512874, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 27 12:41:14 dist-test-slave-bnc1 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512874, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 12:41:14.096156 7604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:14.080073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33427 (local address 127.0.98.60:42401)
0927 12:41:14.080208 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:14.080212 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:14.081940 (+ 1728us) server_negotiation.cc:407] Connection header received
0927 12:41:14.082444 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:14.082448 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:14.082500 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:14.082559 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:14.084035 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:14.084846 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:14.086391 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:14.086738 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:14.086813 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:14.091214 (+ 4401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:14.091218 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:14.091222 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:41:14.091254 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:14.093501 (+ 2247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:14.094225 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:14.094229 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:14.094232 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:14.094315 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:14.094758 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:14.094763 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:14.094767 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:14.095029 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:14.095132 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:14.095838 (+ 706us) server_negotiation.cc:299] Negotiation successful
0927 12:41:14.095975 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 12:41:14.096154 6534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:14.080251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39513 (local address 127.0.98.62:46303)
0927 12:41:14.080368 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:14.080373 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:14.081378 (+ 1005us) server_negotiation.cc:407] Connection header received
0927 12:41:14.082444 (+ 1066us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:14.082448 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:14.082500 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:14.082557 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:14.084034 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:14.084846 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:14.086390 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:14.086738 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:14.086814 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:14.092042 (+ 5228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:14.092047 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:14.092051 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:41:14.092076 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:14.094106 (+ 2030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:14.094570 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:14.094573 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:14.094575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:14.094743 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:14.095306 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:14.095309 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:14.095311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:14.095502 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:14.095581 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:14.095913 (+ 332us) server_negotiation.cc:299] Negotiation successful
0927 12:41:14.096019 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Sep 27 12:41:14 dist-test-slave-bnc1 krb5kdc[6450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512874, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260927 12:41:14.098124 7738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:14.080733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42401 (local address 127.0.98.3:33427)
0927 12:41:14.081849 (+ 1116us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:14.081858 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:14.082222 (+ 364us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:14.082794 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:14.082802 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:14.083253 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:14.083873 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:14.083882 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:14.084989 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:14.084992 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:14.086253 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:14.086263 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:14.087010 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:14.087014 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:14.087326 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:14.088258 (+ 932us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:14.088285 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:14.091014 (+ 2729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:41:14.093674 (+ 2660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:14.093683 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:14.093705 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:14.094106 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:14.094431 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:14.094434 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:14.094436 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:14.094618 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:14.095201 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:41:14.095208 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:41:14.095548 (+ 340us) client_negotiation.cc:769] Sending connection context
0927 12:41:14.096321 (+ 773us) client_negotiation.cc:240] Negotiation successful
0927 12:41:14.096706 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1075,"mutex_wait_us":299,"spinlock_wait_cycles":32512,"thread_start_us":112,"threads_started":1}
I20260927 12:41:14.099865 7733 heartbeater.cc:347] Connected to a master server at 127.0.98.60:42401
I20260927 12:41:14.100138 7733 heartbeater.cc:464] Registering TS with master...
I20260927 12:41:14.100720 7733 heartbeater.cc:511] Master 127.0.98.60:42401 requested a full tablet report, sending...
I20260927 12:41:14.102149 7276 ts_manager.cc:194] Registered new tserver with Master: 849bb514fb834ec38a1f428e351782d0 (127.0.98.3:40415)
I20260927 12:41:14.102253 7217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:14.081547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39971 (local address 127.0.98.61:46395)
0927 12:41:14.081696 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:14.081700 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:14.081716 (+ 16us) server_negotiation.cc:407] Connection header received
0927 12:41:14.082416 (+ 700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:14.082419 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:14.082484 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:14.082591 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:14.084012 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:14.084556 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:14.094050 (+ 9494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:14.094374 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:14.094436 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:14.097729 (+ 3293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:14.097733 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:14.097736 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:41:14.097763 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:14.100181 (+ 2418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:14.100698 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:14.100702 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:14.100703 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:14.100818 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:14.101138 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:14.101142 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:14.101144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:14.101425 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:14.101886 (+ 461us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:14.101973 (+ 87us) server_negotiation.cc:299] Negotiation successful
0927 12:41:14.102086 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":9,"server-negotiator.queue_time_us":69,"spinlock_wait_cycles":5632}
I20260927 12:41:14.102504 7746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:14.087852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55864 (local address 127.0.98.61:46395)
0927 12:41:14.088128 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:14.088133 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:14.089728 (+ 1595us) server_negotiation.cc:407] Connection header received
0927 12:41:14.089767 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:14.089770 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:14.089812 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:14.089879 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:14.090755 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:14.091519 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:14.092713 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:14.093081 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:14.093161 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:14.098126 (+ 4965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:14.098132 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:14.098134 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:41:14.098160 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:14.100172 (+ 2012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:14.100698 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:14.100701 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:14.100703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:14.100827 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:14.101139 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:14.101142 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:14.101143 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:14.101425 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:14.101559 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:14.102218 (+ 659us) server_negotiation.cc:299] Negotiation successful
0927 12:41:14.102338 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":176,"spinlock_wait_cycles":16128,"thread_start_us":86,"threads_started":1}
I20260927 12:41:14.102818 7739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:14.080569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46395 (local address 127.0.98.3:39971)
0927 12:41:14.081638 (+ 1069us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:14.081651 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:14.082219 (+ 568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:14.082794 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:14.082800 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:14.083249 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:14.083860 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:14.083875 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:14.084705 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:14.084708 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:14.093916 (+ 9208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:14.093928 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:14.094595 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:14.094599 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:14.094724 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:14.095951 (+ 1227us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:14.095965 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:14.097546 (+ 1581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:41:14.100300 (+ 2754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:14.100306 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:14.100308 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:14.100580 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:14.100925 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:14.100928 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:14.100930 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:14.101027 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:14.101539 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:41:14.101542 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:41:14.101672 (+ 130us) client_negotiation.cc:769] Sending connection context
0927 12:41:14.101759 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 12:41:14.102603 (+ 844us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1004,"mutex_wait_us":827,"spinlock_wait_cycles":62976,"thread_start_us":162,"threads_started":1}
I20260927 12:41:14.103487 7737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:14.080541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46303 (local address 127.0.98.3:39513)
0927 12:41:14.081225 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:14.081276 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:14.082222 (+ 946us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:14.082794 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:14.082802 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:14.083253 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:14.083875 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:14.083882 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:14.084989 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:14.084992 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:14.086252 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:14.086263 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:14.087010 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:14.087014 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:14.087327 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:14.088258 (+ 931us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:14.088290 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:14.091872 (+ 3582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:41:14.094222 (+ 2350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:14.094224 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:14.094226 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:14.094465 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:14.094850 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:14.094853 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:14.094865 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:14.095203 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:14.095648 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:41:14.095651 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:41:14.095732 (+ 81us) client_negotiation.cc:769] Sending connection context
0927 12:41:14.096349 (+ 617us) client_negotiation.cc:240] Negotiation successful
0927 12:41:14.103307 (+ 6958us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"mutex_wait_us":1203,"spinlock_wait_cycles":18688,"thread_start_us":147,"threads_started":1}
I20260927 12:41:14.103730 7735 heartbeater.cc:347] Connected to a master server at 127.0.98.61:46395
I20260927 12:41:14.103824 7735 heartbeater.cc:464] Registering TS with master...
I20260927 12:41:14.103999 7735 heartbeater.cc:511] Master 127.0.98.61:46395 requested a full tablet report, sending...
I20260927 12:41:14.104507 7734 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46303
I20260927 12:41:14.104600 7734 heartbeater.cc:464] Registering TS with master...
I20260927 12:41:14.104781 7734 heartbeater.cc:511] Master 127.0.98.62:46303 requested a full tablet report, sending...
I20260927 12:41:14.104835 7088 ts_manager.cc:194] Registered new tserver with Master: 849bb514fb834ec38a1f428e351782d0 (127.0.98.3:40415)
I20260927 12:41:14.105639 6495 ts_manager.cc:194] Re-registered known tserver with Master: 849bb514fb834ec38a1f428e351782d0 (127.0.98.3:40415)
I20260927 12:41:14.106498 6495 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39513
I20260927 12:41:14.108700 7463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:14.088428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39348 (local address 127.0.98.60:42401)
0927 12:41:14.088560 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:14.088565 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:14.098707 (+ 10142us) server_negotiation.cc:407] Connection header received
0927 12:41:14.098886 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:14.098889 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:14.098942 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:14.098997 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:14.099850 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:14.100581 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:14.101749 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:14.102043 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:14.102120 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:14.104940 (+ 2820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:14.104944 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:14.104947 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:41:14.104975 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:14.106863 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:14.107377 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:14.107381 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:14.107383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:14.107463 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:14.107745 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:14.107748 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:14.107750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:14.107941 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:14.108048 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:14.108402 (+ 354us) server_negotiation.cc:299] Negotiation successful
0927 12:41:14.108524 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 12:41:14.109807 7744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:14.087129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44622 (local address 127.0.98.62:46303)
0927 12:41:14.088835 (+ 1706us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:14.088839 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:14.089147 (+ 308us) server_negotiation.cc:407] Connection header received
0927 12:41:14.089184 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:14.089186 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:14.089229 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:14.089304 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:14.090388 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:14.091203 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:14.097531 (+ 6328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:14.097900 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:14.097987 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:14.105653 (+ 7666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:14.105657 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:14.105660 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:41:14.105688 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:14.107533 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:14.108016 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:14.108020 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:14.108021 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:14.108106 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:14.108377 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:14.108380 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:14.108381 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:14.108568 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:14.108697 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:14.109465 (+ 768us) server_negotiation.cc:299] Negotiation successful
0927 12:41:14.109682 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1639,"thread_start_us":77,"threads_started":1}
I20260927 12:41:14.112679 6495 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44622:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 12:41:14.115686 6495 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:41:14.118199 7751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42be682b019e44ed8f1a42e5cf28499f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "42be682b019e44ed8f1a42e5cf28499f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "42be682b019e44ed8f1a42e5cf28499f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46303 } } peers { permanent_uuid: "04babc8b97154b25ac1e359afb91c667" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46395 } attrs { promote: false } } peers { permanent_uuid: "aaa07f27c66948e89a3b73053bcaafc0" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 42401 } attrs { promote: true } } }
I20260927 12:41:14.118352 7751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42be682b019e44ed8f1a42e5cf28499f [sys.catalog]: This master's current role is: LEADER
I20260927 12:41:14.121407 7751 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 42be682b019e44ed8f1a42e5cf28499f: attempting to promote NON_VOTER aaa07f27c66948e89a3b73053bcaafc0 to VOTER
I20260927 12:41:14.121753 7751 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42be682b019e44ed8f1a42e5cf28499f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "42be682b019e44ed8f1a42e5cf28499f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46303 } } peers { permanent_uuid: "04babc8b97154b25ac1e359afb91c667" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46395 } attrs { promote: false } } peers { permanent_uuid: "aaa07f27c66948e89a3b73053bcaafc0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42401 } attrs { promote: false } }
I20260927 12:41:14.122790 7105 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667 [term 1 FOLLOWER]: Refusing update from remote peer 42be682b019e44ed8f1a42e5cf28499f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260927 12:41:14.123495 7245 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 42be682b019e44ed8f1a42e5cf28499f [LEADER]: Connected to new peer: Peer: permanent_uuid: "04babc8b97154b25ac1e359afb91c667" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260927 12:41:14.124110 7288 raft_consensus.cc:1261] T 00000000000000000000000000000000 P aaa07f27c66948e89a3b73053bcaafc0 [term 1 LEARNER]: Refusing update from remote peer 42be682b019e44ed8f1a42e5cf28499f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260927 12:41:14.125000 7750 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 42be682b019e44ed8f1a42e5cf28499f [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, aaa07f27c66948e89a3b73053bcaafc0 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "42be682b019e44ed8f1a42e5cf28499f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46303 } } peers { permanent_uuid: "04babc8b97154b25ac1e359afb91c667" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46395 } attrs { promote: false } } peers { permanent_uuid: "aaa07f27c66948e89a3b73053bcaafc0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42401 } attrs { promote: false } } }
I20260927 12:41:14.125154 7245 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 42be682b019e44ed8f1a42e5cf28499f [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaa07f27c66948e89a3b73053bcaafc0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42401 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260927 12:41:14.125698 7105 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, aaa07f27c66948e89a3b73053bcaafc0 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "42be682b019e44ed8f1a42e5cf28499f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46303 } } peers { permanent_uuid: "04babc8b97154b25ac1e359afb91c667" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46395 } attrs { promote: false } } peers { permanent_uuid: "aaa07f27c66948e89a3b73053bcaafc0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42401 } attrs { promote: false } } }
I20260927 12:41:14.126564 7750 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42be682b019e44ed8f1a42e5cf28499f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "42be682b019e44ed8f1a42e5cf28499f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "42be682b019e44ed8f1a42e5cf28499f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46303 } } peers { permanent_uuid: "04babc8b97154b25ac1e359afb91c667" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46395 } attrs { promote: false } } peers { permanent_uuid: "aaa07f27c66948e89a3b73053bcaafc0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42401 } attrs { promote: false } } }
I20260927 12:41:14.126660 7750 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42be682b019e44ed8f1a42e5cf28499f [sys.catalog]: This master's current role is: LEADER
I20260927 12:41:14.126775 7750 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42be682b019e44ed8f1a42e5cf28499f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "42be682b019e44ed8f1a42e5cf28499f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "42be682b019e44ed8f1a42e5cf28499f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46303 } } peers { permanent_uuid: "04babc8b97154b25ac1e359afb91c667" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46395 } attrs { promote: false } } peers { permanent_uuid: "aaa07f27c66948e89a3b73053bcaafc0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42401 } attrs { promote: false } } }
I20260927 12:41:14.126864 7750 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42be682b019e44ed8f1a42e5cf28499f [sys.catalog]: This master's current role is: LEADER
I20260927 12:41:14.126557 7288 raft_consensus.cc:2932] T 00000000000000000000000000000000 P aaa07f27c66948e89a3b73053bcaafc0 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, aaa07f27c66948e89a3b73053bcaafc0 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "42be682b019e44ed8f1a42e5cf28499f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46303 } } peers { permanent_uuid: "04babc8b97154b25ac1e359afb91c667" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46395 } attrs { promote: false } } peers { permanent_uuid: "aaa07f27c66948e89a3b73053bcaafc0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42401 } attrs { promote: false } } }
I20260927 12:41:14.127390 7759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "42be682b019e44ed8f1a42e5cf28499f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "42be682b019e44ed8f1a42e5cf28499f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46303 } } peers { permanent_uuid: "04babc8b97154b25ac1e359afb91c667" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46395 } attrs { promote: false } } peers { permanent_uuid: "aaa07f27c66948e89a3b73053bcaafc0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42401 } attrs { promote: false } } }
I20260927 12:41:14.127507 7759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 12:41:14.128388 7319 sys_catalog.cc:455] T 00000000000000000000000000000000 P aaa07f27c66948e89a3b73053bcaafc0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "42be682b019e44ed8f1a42e5cf28499f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "42be682b019e44ed8f1a42e5cf28499f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46303 } } peers { permanent_uuid: "04babc8b97154b25ac1e359afb91c667" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46395 } attrs { promote: false } } peers { permanent_uuid: "aaa07f27c66948e89a3b73053bcaafc0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42401 } attrs { promote: false } } }
I20260927 12:41:14.128504 7319 sys_catalog.cc:458] T 00000000000000000000000000000000 P aaa07f27c66948e89a3b73053bcaafc0 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 12:41:14.129076 7245 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42be682b019e44ed8f1a42e5cf28499f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "42be682b019e44ed8f1a42e5cf28499f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "42be682b019e44ed8f1a42e5cf28499f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46303 } } peers { permanent_uuid: "04babc8b97154b25ac1e359afb91c667" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46395 } attrs { promote: false } } peers { permanent_uuid: "aaa07f27c66948e89a3b73053bcaafc0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42401 } attrs { promote: false } } }
I20260927 12:41:14.129191 7245 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42be682b019e44ed8f1a42e5cf28499f [sys.catalog]: This master's current role is: LEADER
I20260927 12:41:14.136711 7246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:14.131100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42501 (local address 127.0.0.1:36212)
0927 12:41:14.131231 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:14.131246 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:14.131323 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:14.132846 (+ 1523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:14.132863 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:14.132878 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:41:14.132992 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:14.132999 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:14.134109 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:14.134113 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:14.135071 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:14.135079 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:14.136302 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:14.136306 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:14.136400 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:14.136437 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 12:41:14.136503 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 12:41:14.136577 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":35}
I20260927 12:41:14.136976 7760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:14.131500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40415 (local address 127.0.0.1:41044)
0927 12:41:14.131759 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:14.131773 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:14.131861 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:14.133096 (+ 1235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:14.133098 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:14.133110 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:41:14.133164 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:14.133167 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:14.134308 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:14.134311 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:14.135439 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:14.135450 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:14.136474 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:14.136477 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:14.136651 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:14.136678 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 12:41:14.136752 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 12:41:14.136826 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":124,"threads_started":1}
I20260927 12:41:14.137046 7762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:14.131708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41044 (local address 127.0.98.3:40415)
0927 12:41:14.132329 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:14.132343 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:14.132366 (+ 23us) server_negotiation.cc:407] Connection header received
0927 12:41:14.132441 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:14.132446 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:14.132694 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:14.132882 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:41:14.133274 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:14.134195 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:14.135566 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:14.136355 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:14.136457 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:14.136603 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:14.136807 (+ 204us) server_negotiation.cc:299] Negotiation successful
0927 12:41:14.136916 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":109,"threads_started":1}
I20260927 12:41:14.137514 7761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:14.131577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36212 (local address 127.0.98.1:42501)
0927 12:41:14.132090 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:14.132101 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:14.132126 (+ 25us) server_negotiation.cc:407] Connection header received
0927 12:41:14.132480 (+ 354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:14.132492 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:14.132704 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:14.132883 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:41:14.133086 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:14.134000 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:14.135221 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:14.135962 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:14.136063 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:14.136226 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:14.137322 (+ 1096us) server_negotiation.cc:299] Negotiation successful
0927 12:41:14.137406 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":129,"threads_started":1}
I20260927 12:41:14.139467 7763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:14.131861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43043 (local address 127.0.0.1:37330)
0927 12:41:14.134199 (+ 2338us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:14.134210 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:14.134302 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:14.135034 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:14.135037 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:14.135055 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:41:14.135156 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:14.135164 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:14.136599 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:14.136602 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:14.137928 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:14.137941 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:14.139038 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:14.139041 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:14.139126 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:14.139164 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 12:41:14.139214 (+ 50us) client_negotiation.cc:240] Negotiation successful
0927 12:41:14.139288 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2284,"thread_start_us":74,"threads_started":1}
I20260927 12:41:14.140445 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:14.132206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37330 (local address 127.0.98.2:43043)
0927 12:41:14.134552 (+ 2346us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:14.134567 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:14.134589 (+ 22us) server_negotiation.cc:407] Connection header received
0927 12:41:14.134665 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:14.134670 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:14.134896 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:14.135044 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:41:14.135246 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:14.136478 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:14.138096 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:14.138896 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:14.139862 (+ 966us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:14.140135 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:14.140209 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 12:41:14.140306 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2239,"thread_start_us":87,"threads_started":1}
I20260927 12:41:14.140296 7667 tablet_service.cc:1467] Processing CreateTablet for tablet 0faa50c126ce458897d137327f255583 (DEFAULT_TABLE table=test-table [id=7d2d01391345467ba65642f26c7212fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:41:14.140584 7393 tablet_service.cc:1467] Processing CreateTablet for tablet 0faa50c126ce458897d137327f255583 (DEFAULT_TABLE table=test-table [id=7d2d01391345467ba65642f26c7212fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:41:14.141414 7667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0faa50c126ce458897d137327f255583. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:41:14.141567 7393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0faa50c126ce458897d137327f255583. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:41:14.143460 7530 tablet_service.cc:1467] Processing CreateTablet for tablet 0faa50c126ce458897d137327f255583 (DEFAULT_TABLE table=test-table [id=7d2d01391345467ba65642f26c7212fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:41:14.144513 7530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0faa50c126ce458897d137327f255583. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:41:14.148761 7765 tablet_bootstrap.cc:492] T 0faa50c126ce458897d137327f255583 P 849bb514fb834ec38a1f428e351782d0: Bootstrap starting.
I20260927 12:41:14.151160 7765 tablet_bootstrap.cc:654] T 0faa50c126ce458897d137327f255583 P 849bb514fb834ec38a1f428e351782d0: Neither blocks nor log segments found. Creating new log.
I20260927 12:41:14.151875 7765 log.cc:824] T 0faa50c126ce458897d137327f255583 P 849bb514fb834ec38a1f428e351782d0: Log is configured to *not* fsync() on all Append() calls
I20260927 12:41:14.151984 7767 tablet_bootstrap.cc:492] T 0faa50c126ce458897d137327f255583 P 9fd3b55ca4154ec69ee88b69c88fb053: Bootstrap starting.
I20260927 12:41:14.152554 7766 tablet_bootstrap.cc:492] T 0faa50c126ce458897d137327f255583 P 4ac28926eacd419bad1e9f8b5e063a36: Bootstrap starting.
I20260927 12:41:14.154125 7767 tablet_bootstrap.cc:654] T 0faa50c126ce458897d137327f255583 P 9fd3b55ca4154ec69ee88b69c88fb053: Neither blocks nor log segments found. Creating new log.
I20260927 12:41:14.155027 7766 tablet_bootstrap.cc:654] T 0faa50c126ce458897d137327f255583 P 4ac28926eacd419bad1e9f8b5e063a36: Neither blocks nor log segments found. Creating new log.
I20260927 12:41:14.155172 7765 tablet_bootstrap.cc:492] T 0faa50c126ce458897d137327f255583 P 849bb514fb834ec38a1f428e351782d0: No bootstrap required, opened a new log
I20260927 12:41:14.155375 7765 ts_tablet_manager.cc:1397] T 0faa50c126ce458897d137327f255583 P 849bb514fb834ec38a1f428e351782d0: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260927 12:41:14.155414 7767 log.cc:824] T 0faa50c126ce458897d137327f255583 P 9fd3b55ca4154ec69ee88b69c88fb053: Log is configured to *not* fsync() on all Append() calls
I20260927 12:41:14.155849 7766 log.cc:824] T 0faa50c126ce458897d137327f255583 P 4ac28926eacd419bad1e9f8b5e063a36: Log is configured to *not* fsync() on all Append() calls
I20260927 12:41:14.157553 7767 tablet_bootstrap.cc:492] T 0faa50c126ce458897d137327f255583 P 9fd3b55ca4154ec69ee88b69c88fb053: No bootstrap required, opened a new log
I20260927 12:41:14.157779 7767 ts_tablet_manager.cc:1397] T 0faa50c126ce458897d137327f255583 P 9fd3b55ca4154ec69ee88b69c88fb053: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260927 12:41:14.157882 7766 tablet_bootstrap.cc:492] T 0faa50c126ce458897d137327f255583 P 4ac28926eacd419bad1e9f8b5e063a36: No bootstrap required, opened a new log
I20260927 12:41:14.158069 7766 ts_tablet_manager.cc:1397] T 0faa50c126ce458897d137327f255583 P 4ac28926eacd419bad1e9f8b5e063a36: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260927 12:41:14.159835 7765 raft_consensus.cc:348] T 0faa50c126ce458897d137327f255583 P 849bb514fb834ec38a1f428e351782d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ac28926eacd419bad1e9f8b5e063a36" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42501 } } peers { permanent_uuid: "849bb514fb834ec38a1f428e351782d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40415 } } peers { permanent_uuid: "9fd3b55ca4154ec69ee88b69c88fb053" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43043 } }
I20260927 12:41:14.160104 7765 raft_consensus.cc:374] T 0faa50c126ce458897d137327f255583 P 849bb514fb834ec38a1f428e351782d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:41:14.160158 7765 raft_consensus.cc:729] T 0faa50c126ce458897d137327f255583 P 849bb514fb834ec38a1f428e351782d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 849bb514fb834ec38a1f428e351782d0, State: Initialized, Role: FOLLOWER
I20260927 12:41:14.160606 7765 consensus_queue.cc:260] T 0faa50c126ce458897d137327f255583 P 849bb514fb834ec38a1f428e351782d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ac28926eacd419bad1e9f8b5e063a36" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42501 } } peers { permanent_uuid: "849bb514fb834ec38a1f428e351782d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40415 } } peers { permanent_uuid: "9fd3b55ca4154ec69ee88b69c88fb053" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43043 } }
I20260927 12:41:14.161291 7734 heartbeater.cc:503] Master 127.0.98.62:46303 was elected leader, sending a full tablet report...
I20260927 12:41:14.161628 7765 ts_tablet_manager.cc:1428] T 0faa50c126ce458897d137327f255583 P 849bb514fb834ec38a1f428e351782d0: Time spent starting tablet: real 0.006s user 0.006s sys 0.002s
I20260927 12:41:14.163424 7766 raft_consensus.cc:348] T 0faa50c126ce458897d137327f255583 P 4ac28926eacd419bad1e9f8b5e063a36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ac28926eacd419bad1e9f8b5e063a36" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42501 } } peers { permanent_uuid: "849bb514fb834ec38a1f428e351782d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40415 } } peers { permanent_uuid: "9fd3b55ca4154ec69ee88b69c88fb053" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43043 } }
I20260927 12:41:14.163681 7766 raft_consensus.cc:374] T 0faa50c126ce458897d137327f255583 P 4ac28926eacd419bad1e9f8b5e063a36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:41:14.163736 7766 raft_consensus.cc:729] T 0faa50c126ce458897d137327f255583 P 4ac28926eacd419bad1e9f8b5e063a36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ac28926eacd419bad1e9f8b5e063a36, State: Initialized, Role: FOLLOWER
I20260927 12:41:14.163381 7767 raft_consensus.cc:348] T 0faa50c126ce458897d137327f255583 P 9fd3b55ca4154ec69ee88b69c88fb053 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ac28926eacd419bad1e9f8b5e063a36" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42501 } } peers { permanent_uuid: "849bb514fb834ec38a1f428e351782d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40415 } } peers { permanent_uuid: "9fd3b55ca4154ec69ee88b69c88fb053" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43043 } }
I20260927 12:41:14.163874 7767 raft_consensus.cc:374] T 0faa50c126ce458897d137327f255583 P 9fd3b55ca4154ec69ee88b69c88fb053 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:41:14.163941 7767 raft_consensus.cc:729] T 0faa50c126ce458897d137327f255583 P 9fd3b55ca4154ec69ee88b69c88fb053 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fd3b55ca4154ec69ee88b69c88fb053, State: Initialized, Role: FOLLOWER
I20260927 12:41:14.164153 7766 consensus_queue.cc:260] T 0faa50c126ce458897d137327f255583 P 4ac28926eacd419bad1e9f8b5e063a36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ac28926eacd419bad1e9f8b5e063a36" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42501 } } peers { permanent_uuid: "849bb514fb834ec38a1f428e351782d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40415 } } peers { permanent_uuid: "9fd3b55ca4154ec69ee88b69c88fb053" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43043 } }
I20260927 12:41:14.164376 7767 consensus_queue.cc:260] T 0faa50c126ce458897d137327f255583 P 9fd3b55ca4154ec69ee88b69c88fb053 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ac28926eacd419bad1e9f8b5e063a36" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42501 } } peers { permanent_uuid: "849bb514fb834ec38a1f428e351782d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40415 } } peers { permanent_uuid: "9fd3b55ca4154ec69ee88b69c88fb053" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43043 } }
I20260927 12:41:14.164888 7460 heartbeater.cc:503] Master 127.0.98.62:46303 was elected leader, sending a full tablet report...
I20260927 12:41:14.165199 7766 ts_tablet_manager.cc:1428] T 0faa50c126ce458897d137327f255583 P 4ac28926eacd419bad1e9f8b5e063a36: Time spent starting tablet: real 0.007s user 0.004s sys 0.002s
I20260927 12:41:14.165397 7767 ts_tablet_manager.cc:1428] T 0faa50c126ce458897d137327f255583 P 9fd3b55ca4154ec69ee88b69c88fb053: Time spent starting tablet: real 0.008s user 0.001s sys 0.004s
I20260927 12:41:14.168527 7597 heartbeater.cc:503] Master 127.0.98.62:46303 was elected leader, sending a full tablet report...
W20260927 12:41:14.180598 7462 tablet.cc:2369] T 0faa50c126ce458897d137327f255583 P 4ac28926eacd419bad1e9f8b5e063a36: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:41:14.249694 7773 raft_consensus.cc:482] T 0faa50c126ce458897d137327f255583 P 9fd3b55ca4154ec69ee88b69c88fb053 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:41:14.249917 7773 raft_consensus.cc:504] T 0faa50c126ce458897d137327f255583 P 9fd3b55ca4154ec69ee88b69c88fb053 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ac28926eacd419bad1e9f8b5e063a36" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42501 } } peers { permanent_uuid: "849bb514fb834ec38a1f428e351782d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40415 } } peers { permanent_uuid: "9fd3b55ca4154ec69ee88b69c88fb053" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43043 } }
I20260927 12:41:14.251266 7773 leader_election.cc:288] T 0faa50c126ce458897d137327f255583 P 9fd3b55ca4154ec69ee88b69c88fb053 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ac28926eacd419bad1e9f8b5e063a36 (127.0.98.1:42501), 849bb514fb834ec38a1f428e351782d0 (127.0.98.3:40415)
I20260927 12:41:14.256445 7762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:14.251903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60443 (local address 127.0.98.3:40415)
0927 12:41:14.252022 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:14.252027 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:14.252040 (+ 13us) server_negotiation.cc:407] Connection header received
0927 12:41:14.252091 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:14.252095 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:14.252145 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:14.252229 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:41:14.252496 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:14.253209 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:14.255013 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:14.255767 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:14.255861 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:14.255916 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:14.256227 (+ 311us) server_negotiation.cc:299] Negotiation successful
0927 12:41:14.256313 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 12:41:14.256449 7761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:14.251557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51865 (local address 127.0.98.1:42501)
0927 12:41:14.251843 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:14.251848 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:14.251864 (+ 16us) server_negotiation.cc:407] Connection header received
0927 12:41:14.251922 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:14.251926 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:14.251980 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:14.252050 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:41:14.252372 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:14.253165 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:14.255014 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:14.255714 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:14.255846 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:14.255916 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:14.256265 (+ 349us) server_negotiation.cc:299] Negotiation successful
0927 12:41:14.256342 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203}
I20260927 12:41:14.256501 7601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:14.251444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42501 (local address 127.0.98.2:51865)
0927 12:41:14.251636 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:14.251652 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:14.251740 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:14.252071 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:14.252074 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:14.252089 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:41:14.252252 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:14.252258 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:14.253304 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:14.253307 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:14.254863 (+ 1556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:14.254869 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:14.255860 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:14.255864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:14.256029 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:14.256105 (+ 76us) client_negotiation.cc:769] Sending connection context
0927 12:41:14.256194 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 12:41:14.256301 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":242}
I20260927 12:41:14.256501 7602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:14.251566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40415 (local address 127.0.98.2:60443)
0927 12:41:14.251742 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:14.251756 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:14.251840 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:14.252257 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:14.252259 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:14.252281 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:41:14.252398 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:14.252403 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:14.253304 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:14.253307 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:14.254858 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:14.254869 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:14.255878 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:14.255881 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:14.256032 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:14.256110 (+ 78us) client_negotiation.cc:769] Sending connection context
0927 12:41:14.256193 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 12:41:14.256301 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"mutex_wait_us":223}
I20260927 12:41:14.257239 7687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0faa50c126ce458897d137327f255583" candidate_uuid: "9fd3b55ca4154ec69ee88b69c88fb053" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "849bb514fb834ec38a1f428e351782d0" is_pre_election: true
I20260927 12:41:14.257304 7413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0faa50c126ce458897d137327f255583" candidate_uuid: "9fd3b55ca4154ec69ee88b69c88fb053" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ac28926eacd419bad1e9f8b5e063a36" is_pre_election: true
I20260927 12:41:14.257586 7687 raft_consensus.cc:2454] T 0faa50c126ce458897d137327f255583 P 849bb514fb834ec38a1f428e351782d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9fd3b55ca4154ec69ee88b69c88fb053 in term 0.
I20260927 12:41:14.257643 7413 raft_consensus.cc:2454] T 0faa50c126ce458897d137327f255583 P 4ac28926eacd419bad1e9f8b5e063a36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9fd3b55ca4154ec69ee88b69c88fb053 in term 0.
I20260927 12:41:14.258106 7486 leader_election.cc:302] T 0faa50c126ce458897d137327f255583 P 9fd3b55ca4154ec69ee88b69c88fb053 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4ac28926eacd419bad1e9f8b5e063a36, 849bb514fb834ec38a1f428e351782d0, 9fd3b55ca4154ec69ee88b69c88fb053; no voters:
I20260927 12:41:14.258399 7773 raft_consensus.cc:2781] T 0faa50c126ce458897d137327f255583 P 9fd3b55ca4154ec69ee88b69c88fb053 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:41:14.258481 7773 raft_consensus.cc:482] T 0faa50c126ce458897d137327f255583 P 9fd3b55ca4154ec69ee88b69c88fb053 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:41:14.258515 7773 raft_consensus.cc:3037] T 0faa50c126ce458897d137327f255583 P 9fd3b55ca4154ec69ee88b69c88fb053 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:41:14.259644 7773 raft_consensus.cc:504] T 0faa50c126ce458897d137327f255583 P 9fd3b55ca4154ec69ee88b69c88fb053 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ac28926eacd419bad1e9f8b5e063a36" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42501 } } peers { permanent_uuid: "849bb514fb834ec38a1f428e351782d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40415 } } peers { permanent_uuid: "9fd3b55ca4154ec69ee88b69c88fb053" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43043 } }
I20260927 12:41:14.260012 7773 leader_election.cc:288] T 0faa50c126ce458897d137327f255583 P 9fd3b55ca4154ec69ee88b69c88fb053 [CANDIDATE]: Term 1 election: Requested vote from peers 4ac28926eacd419bad1e9f8b5e063a36 (127.0.98.1:42501), 849bb514fb834ec38a1f428e351782d0 (127.0.98.3:40415)
I20260927 12:41:14.260458 7687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0faa50c126ce458897d137327f255583" candidate_uuid: "9fd3b55ca4154ec69ee88b69c88fb053" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "849bb514fb834ec38a1f428e351782d0"
I20260927 12:41:14.260458 7413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0faa50c126ce458897d137327f255583" candidate_uuid: "9fd3b55ca4154ec69ee88b69c88fb053" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ac28926eacd419bad1e9f8b5e063a36"
I20260927 12:41:14.260591 7687 raft_consensus.cc:3037] T 0faa50c126ce458897d137327f255583 P 849bb514fb834ec38a1f428e351782d0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:41:14.260591 7413 raft_consensus.cc:3037] T 0faa50c126ce458897d137327f255583 P 4ac28926eacd419bad1e9f8b5e063a36 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:41:14.261888 7413 raft_consensus.cc:2454] T 0faa50c126ce458897d137327f255583 P 4ac28926eacd419bad1e9f8b5e063a36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fd3b55ca4154ec69ee88b69c88fb053 in term 1.
I20260927 12:41:14.261888 7687 raft_consensus.cc:2454] T 0faa50c126ce458897d137327f255583 P 849bb514fb834ec38a1f428e351782d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fd3b55ca4154ec69ee88b69c88fb053 in term 1.
I20260927 12:41:14.262176 7486 leader_election.cc:302] T 0faa50c126ce458897d137327f255583 P 9fd3b55ca4154ec69ee88b69c88fb053 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ac28926eacd419bad1e9f8b5e063a36, 9fd3b55ca4154ec69ee88b69c88fb053; no voters:
I20260927 12:41:14.262348 7773 raft_consensus.cc:2781] T 0faa50c126ce458897d137327f255583 P 9fd3b55ca4154ec69ee88b69c88fb053 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:41:14.262596 7773 raft_consensus.cc:686] T 0faa50c126ce458897d137327f255583 P 9fd3b55ca4154ec69ee88b69c88fb053 [term 1 LEADER]: Becoming Leader. State: Replica: 9fd3b55ca4154ec69ee88b69c88fb053, State: Running, Role: LEADER
I20260927 12:41:14.262948 7773 consensus_queue.cc:237] T 0faa50c126ce458897d137327f255583 P 9fd3b55ca4154ec69ee88b69c88fb053 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ac28926eacd419bad1e9f8b5e063a36" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42501 } } peers { permanent_uuid: "849bb514fb834ec38a1f428e351782d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40415 } } peers { permanent_uuid: "9fd3b55ca4154ec69ee88b69c88fb053" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43043 } }
I20260927 12:41:14.266340 6493 catalog_manager.cc:5495] T 0faa50c126ce458897d137327f255583 P 9fd3b55ca4154ec69ee88b69c88fb053 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9fd3b55ca4154ec69ee88b69c88fb053 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "9fd3b55ca4154ec69ee88b69c88fb053" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ac28926eacd419bad1e9f8b5e063a36" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "849bb514fb834ec38a1f428e351782d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9fd3b55ca4154ec69ee88b69c88fb053" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43043 } health_report { overall_health: HEALTHY } } }
I20260927 12:41:14.289570 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:14.286325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37332 (local address 127.0.98.2:43043)
0927 12:41:14.286449 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:14.286453 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:14.286467 (+ 14us) server_negotiation.cc:407] Connection header received
0927 12:41:14.286546 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:14.286548 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:14.286604 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:14.286698 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:41:14.286926 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:14.287422 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:14.288205 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:14.288410 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:14.288486 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:14.288745 (+ 259us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:41:14.289170 (+ 425us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:41:14.289279 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:14.289331 (+ 52us) server_negotiation.cc:299] Negotiation successful
0927 12:41:14.289383 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 12:41:14.295567 7687 raft_consensus.cc:1261] T 0faa50c126ce458897d137327f255583 P 849bb514fb834ec38a1f428e351782d0 [term 1 FOLLOWER]: Refusing update from remote peer 9fd3b55ca4154ec69ee88b69c88fb053: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:41:14.295567 7413 raft_consensus.cc:1261] T 0faa50c126ce458897d137327f255583 P 4ac28926eacd419bad1e9f8b5e063a36 [term 1 FOLLOWER]: Refusing update from remote peer 9fd3b55ca4154ec69ee88b69c88fb053: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:41:14.296243 7773 consensus_queue.cc:1035] T 0faa50c126ce458897d137327f255583 P 9fd3b55ca4154ec69ee88b69c88fb053 [LEADER]: Connected to new peer: Peer: permanent_uuid: "849bb514fb834ec38a1f428e351782d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:41:14.296458 7774 consensus_queue.cc:1035] T 0faa50c126ce458897d137327f255583 P 9fd3b55ca4154ec69ee88b69c88fb053 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ac28926eacd419bad1e9f8b5e063a36" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:41:14.304185 7777 mvcc.cc:205] Tried to move back new op lower bound from 7333940733106253824 to 7333940732982931456. Current Snapshot: MvccSnapshot[applied={T|T < 7333940733106253824}]
I20260927 12:41:14.316011 6493 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:41:14.316143 6493 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:41:14.320947 6493 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 42be682b019e44ed8f1a42e5cf28499f: Sending DeleteTablet for 3 replicas of tablet 0faa50c126ce458897d137327f255583
I20260927 12:41:14.321781 7530 tablet_service.cc:1514] Processing DeleteTablet for tablet 0faa50c126ce458897d137327f255583 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:41:14 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37330
I20260927 12:41:14.321894 7393 tablet_service.cc:1514] Processing DeleteTablet for tablet 0faa50c126ce458897d137327f255583 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:41:14 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36212
I20260927 12:41:14.322357 7667 tablet_service.cc:1514] Processing DeleteTablet for tablet 0faa50c126ce458897d137327f255583 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:41:14 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41044
I20260927 12:41:14.322963 7790 tablet_replica.cc:330] T 0faa50c126ce458897d137327f255583 P 849bb514fb834ec38a1f428e351782d0: stopping tablet replica
I20260927 12:41:14.323326 7790 raft_consensus.cc:2229] T 0faa50c126ce458897d137327f255583 P 849bb514fb834ec38a1f428e351782d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:41:14.323731 7790 raft_consensus.cc:2258] T 0faa50c126ce458897d137327f255583 P 849bb514fb834ec38a1f428e351782d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:41:14.324055 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 6463
I20260927 12:41:14.324765 7787 tablet_replica.cc:330] T 0faa50c126ce458897d137327f255583 P 9fd3b55ca4154ec69ee88b69c88fb053: stopping tablet replica
I20260927 12:41:14.325052 7787 raft_consensus.cc:2229] T 0faa50c126ce458897d137327f255583 P 9fd3b55ca4154ec69ee88b69c88fb053 [term 1 LEADER]: Raft consensus shutting down.
I20260927 12:41:14.325466 7787 raft_consensus.cc:2258] T 0faa50c126ce458897d137327f255583 P 9fd3b55ca4154ec69ee88b69c88fb053 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:41:14.325965 7790 ts_tablet_manager.cc:1905] T 0faa50c126ce458897d137327f255583 P 849bb514fb834ec38a1f428e351782d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:41:14.328635 7790 ts_tablet_manager.cc:1918] T 0faa50c126ce458897d137327f255583 P 849bb514fb834ec38a1f428e351782d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:41:14.328792 7790 log.cc:1196] T 0faa50c126ce458897d137327f255583 P 849bb514fb834ec38a1f428e351782d0: Deleting WAL directory at /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-2/wal/wals/0faa50c126ce458897d137327f255583
I20260927 12:41:14.329177 7790 ts_tablet_manager.cc:1939] T 0faa50c126ce458897d137327f255583 P 849bb514fb834ec38a1f428e351782d0: Deleting consensus metadata
W20260927 12:41:14.332618 7620 connection.cc:535] server connection from 127.0.0.1:41044 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 12:41:14.333446 7788 tablet_replica.cc:330] T 0faa50c126ce458897d137327f255583 P 4ac28926eacd419bad1e9f8b5e063a36: stopping tablet replica
I20260927 12:41:14.333793 7788 raft_consensus.cc:2229] T 0faa50c126ce458897d137327f255583 P 4ac28926eacd419bad1e9f8b5e063a36 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:41:14.334144 7788 raft_consensus.cc:2258] T 0faa50c126ce458897d137327f255583 P 4ac28926eacd419bad1e9f8b5e063a36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:41:14.334555 7787 ts_tablet_manager.cc:1905] T 0faa50c126ce458897d137327f255583 P 9fd3b55ca4154ec69ee88b69c88fb053: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:41:14.334837 7746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:14.323168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55870 (local address 127.0.98.61:46395)
0927 12:41:14.323278 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:14.323283 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:14.323307 (+ 24us) server_negotiation.cc:407] Connection header received
0927 12:41:14.323342 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:14.323345 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:14.323386 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:14.323457 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:14.325242 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:14.326147 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:14.327630 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:14.328067 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:14.328179 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:14.330478 (+ 2299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:14.330484 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:14.330491 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:41:14.330521 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:14.332367 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:14.333423 (+ 1056us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:14.333426 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:14.333428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:14.333517 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:14.333849 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:14.333853 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:14.333854 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:14.334064 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:14.334169 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:14.334552 (+ 383us) server_negotiation.cc:299] Negotiation successful
0927 12:41:14.334685 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 12:41:14.337502 7787 ts_tablet_manager.cc:1918] T 0faa50c126ce458897d137327f255583 P 9fd3b55ca4154ec69ee88b69c88fb053: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:41:14.337669 7787 log.cc:1196] T 0faa50c126ce458897d137327f255583 P 9fd3b55ca4154ec69ee88b69c88fb053: Deleting WAL directory at /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-1/wal/wals/0faa50c126ce458897d137327f255583
I20260927 12:41:14.337697 7788 ts_tablet_manager.cc:1905] T 0faa50c126ce458897d137327f255583 P 4ac28926eacd419bad1e9f8b5e063a36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:41:14.338089 7787 ts_tablet_manager.cc:1939] T 0faa50c126ce458897d137327f255583 P 9fd3b55ca4154ec69ee88b69c88fb053: Deleting consensus metadata
W20260927 12:41:14.339128 7485 connection.cc:441] server connection from 127.0.0.1:37330 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37330 (request call id 1) could send its response
I20260927 12:41:14.340328 7788 ts_tablet_manager.cc:1918] T 0faa50c126ce458897d137327f255583 P 4ac28926eacd419bad1e9f8b5e063a36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:41:14.340432 7788 log.cc:1196] T 0faa50c126ce458897d137327f255583 P 4ac28926eacd419bad1e9f8b5e063a36: Deleting WAL directory at /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-0/wal/wals/0faa50c126ce458897d137327f255583
I20260927 12:41:14.340786 7788 ts_tablet_manager.cc:1939] T 0faa50c126ce458897d137327f255583 P 4ac28926eacd419bad1e9f8b5e063a36: Deleting consensus metadata
W20260927 12:41:14.342021 7346 connection.cc:441] server connection from 127.0.0.1:36212 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36212 (request call id 1) could send its response
I20260927 12:41:14.345556 7463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:14.323167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39350 (local address 127.0.98.60:42401)
0927 12:41:14.323599 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:14.323603 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:14.336226 (+ 12623us) server_negotiation.cc:407] Connection header received
0927 12:41:14.336409 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:14.336414 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:14.336473 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:14.336559 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:14.337429 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:14.338214 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:14.339600 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:14.339912 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:14.339994 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:14.341811 (+ 1817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:14.341815 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:14.341818 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:41:14.341844 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:14.343500 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:14.344038 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:14.344043 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:14.344045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:14.344132 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:14.344451 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:14.344454 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:14.344456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:14.344699 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:14.344799 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:14.345211 (+ 412us) server_negotiation.cc:299] Negotiation successful
0927 12:41:14.345374 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":363}
I20260927 12:41:14.352672 7604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:14.341906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39362 (local address 127.0.98.60:42401)
0927 12:41:14.342019 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:14.342023 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:14.343240 (+ 1217us) server_negotiation.cc:407] Connection header received
0927 12:41:14.343427 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:14.343430 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:14.343481 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:14.343533 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:14.344472 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:14.345319 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:14.347015 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:14.347390 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:14.347493 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:14.349085 (+ 1592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:14.349088 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:14.349093 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:41:14.349127 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:14.350967 (+ 1840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:14.351363 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:14.351368 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:14.351369 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:14.351450 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:14.351735 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:14.351741 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:14.351743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:14.351952 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:14.352058 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:14.352398 (+ 340us) server_negotiation.cc:299] Negotiation successful
0927 12:41:14.352518 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":44}
I20260927 12:41:14.353021 7746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:14.342842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55886 (local address 127.0.98.61:46395)
0927 12:41:14.342947 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:14.342951 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:14.344641 (+ 1690us) server_negotiation.cc:407] Connection header received
0927 12:41:14.344818 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:14.344824 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:14.344878 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:14.344929 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:14.345772 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:14.346496 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:14.347680 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:14.348025 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:14.348132 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:14.349518 (+ 1386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:14.349522 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:14.349525 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:41:14.349550 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:14.351364 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:14.351785 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:14.351788 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:14.351790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:14.351872 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:14.352159 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:14.352162 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:14.352163 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:14.352366 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:14.352481 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:14.352801 (+ 320us) server_negotiation.cc:299] Negotiation successful
0927 12:41:14.352910 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 12:41:15.308003 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:15.307286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46303 (local address 127.0.98.2:37247)
0927 12:41:15.307795 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:15.307893 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":77,"threads_started":1}
I20260927 12:41:15.308308 7798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:15.307841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46303 (local address 127.0.98.1:54361)
0927 12:41:15.308113 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:15.308195 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":62,"threads_started":1}
W20260927 12:41:15.308389 7597 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46303 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:46303: connect: Connection refused (error 111)
W20260927 12:41:15.308600 7460 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46303 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:46303: connect: Connection refused (error 111)
I20260927 12:41:15.309480 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:15.309138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46303 (local address 127.0.98.3:50983)
0927 12:41:15.309337 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:15.309409 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":70,"threads_started":1}
W20260927 12:41:15.309756 7734 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46303 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:46303: connect: Connection refused (error 111)
I20260927 12:41:15.820787 7800 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 42be682b019e44ed8f1a42e5cf28499f)
I20260927 12:41:15.820832 7801 raft_consensus.cc:482] T 00000000000000000000000000000000 P aaa07f27c66948e89a3b73053bcaafc0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 42be682b019e44ed8f1a42e5cf28499f)
I20260927 12:41:15.821024 7800 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "42be682b019e44ed8f1a42e5cf28499f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46303 } } peers { permanent_uuid: "04babc8b97154b25ac1e359afb91c667" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46395 } attrs { promote: false } } peers { permanent_uuid: "aaa07f27c66948e89a3b73053bcaafc0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42401 } attrs { promote: false } }
I20260927 12:41:15.821015 7801 raft_consensus.cc:504] T 00000000000000000000000000000000 P aaa07f27c66948e89a3b73053bcaafc0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "42be682b019e44ed8f1a42e5cf28499f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46303 } } peers { permanent_uuid: "04babc8b97154b25ac1e359afb91c667" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46395 } attrs { promote: false } } peers { permanent_uuid: "aaa07f27c66948e89a3b73053bcaafc0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42401 } attrs { promote: false } }
I20260927 12:41:15.822046 7801 leader_election.cc:288] T 00000000000000000000000000000000 P aaa07f27c66948e89a3b73053bcaafc0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42be682b019e44ed8f1a42e5cf28499f (127.0.98.62:46303), 04babc8b97154b25ac1e359afb91c667 (127.0.98.61:46395)
I20260927 12:41:15.822470 7800 leader_election.cc:288] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42be682b019e44ed8f1a42e5cf28499f (127.0.98.62:46303), aaa07f27c66948e89a3b73053bcaafc0 (127.0.98.60:42401)
I20260927 12:41:15.822690 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:15.822225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46303 (local address 127.0.0.1:45098)
0927 12:41:15.822510 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:15.822579 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":82,"threads_started":1}
I20260927 12:41:15.822958 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:15.822176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46303 (local address 127.0.0.1:45100)
0927 12:41:15.822783 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:15.822858 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":70,"threads_started":1}
W20260927 12:41:15.822953 7082 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46303: connect: Connection refused (error 111)
W20260927 12:41:15.823199 7267 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46303: connect: Connection refused (error 111)
I20260927 12:41:15.823777 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:15.823547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46303 (local address 127.0.0.1:45116)
0927 12:41:15.823659 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:15.823701 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260927 12:41:15.823978 7082 leader_election.cc:334] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 42be682b019e44ed8f1a42e5cf28499f (127.0.98.62:46303): Network error: Client connection negotiation failed: client connection to 127.0.98.62:46303: connect: Connection refused (error 111)
I20260927 12:41:15.824126 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:15.823905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46303 (local address 127.0.0.1:45126)
0927 12:41:15.824025 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:15.824067 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260927 12:41:15.824256 7267 leader_election.cc:334] T 00000000000000000000000000000000 P aaa07f27c66948e89a3b73053bcaafc0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 42be682b019e44ed8f1a42e5cf28499f (127.0.98.62:46303): Network error: Client connection negotiation failed: client connection to 127.0.98.62:46303: connect: Connection refused (error 111)
I20260927 12:41:15.826383 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:15.822740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39368 (local address 127.0.98.60:42401)
0927 12:41:15.822928 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:15.822932 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:15.822946 (+ 14us) server_negotiation.cc:407] Connection header received
0927 12:41:15.822995 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:15.822998 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:15.823041 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:15.823127 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:41:15.823531 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:15.824337 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:15.825342 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:15.825802 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:15.825942 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:15.826038 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:15.826173 (+ 135us) server_negotiation.cc:299] Negotiation successful
0927 12:41:15.826258 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":134,"thread_start_us":56,"threads_started":1}
I20260927 12:41:15.826417 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:15.822402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55900 (local address 127.0.98.61:46395)
0927 12:41:15.822764 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:15.822768 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:15.823053 (+ 285us) server_negotiation.cc:407] Connection header received
0927 12:41:15.823095 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:15.823100 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:15.823144 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:15.823213 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:41:15.823591 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:15.824348 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:15.825342 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:15.825812 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:15.825953 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:15.826012 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:15.826208 (+ 196us) server_negotiation.cc:299] Negotiation successful
0927 12:41:15.826294 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":245,"server-negotiator.queue_time_us":312,"spinlock_wait_cycles":1408,"thread_start_us":45,"threads_started":1}
I20260927 12:41:15.826613 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:15.822176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42401 (local address 127.0.0.1:39368)
0927 12:41:15.822458 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:15.822473 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:15.822588 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:15.823255 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:15.823258 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:15.823283 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:41:15.823405 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:15.823412 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:15.824431 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:15.824437 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:15.825220 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:15.825228 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:15.825938 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:15.825942 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:15.826034 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:15.826063 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 12:41:15.826375 (+ 312us) client_negotiation.cc:240] Negotiation successful
0927 12:41:15.826469 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"spinlock_wait_cycles":896,"thread_start_us":63,"threads_started":1}
I20260927 12:41:15.826629 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:15.822566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46395 (local address 127.0.0.1:55900)
0927 12:41:15.822868 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:15.822879 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:15.822966 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:15.823332 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:15.823335 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:15.823352 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:41:15.823451 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:15.823457 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:15.824481 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:15.824483 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:15.825216 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:15.825224 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:15.825955 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:15.825959 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:15.826069 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:15.826086 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 12:41:15.826407 (+ 321us) client_negotiation.cc:240] Negotiation successful
0927 12:41:15.826497 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":4,"thread_start_us":45,"threads_started":1}
I20260927 12:41:15.827040 7288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "04babc8b97154b25ac1e359afb91c667" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "aaa07f27c66948e89a3b73053bcaafc0" is_pre_election: true
I20260927 12:41:15.827069 7105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "aaa07f27c66948e89a3b73053bcaafc0" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "04babc8b97154b25ac1e359afb91c667" is_pre_election: true
I20260927 12:41:15.827226 7288 raft_consensus.cc:2454] T 00000000000000000000000000000000 P aaa07f27c66948e89a3b73053bcaafc0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04babc8b97154b25ac1e359afb91c667 in term 1.
I20260927 12:41:15.827231 7105 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aaa07f27c66948e89a3b73053bcaafc0 in term 1.
I20260927 12:41:15.827566 7266 leader_election.cc:302] T 00000000000000000000000000000000 P aaa07f27c66948e89a3b73053bcaafc0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 04babc8b97154b25ac1e359afb91c667, aaa07f27c66948e89a3b73053bcaafc0; no voters: 42be682b019e44ed8f1a42e5cf28499f
I20260927 12:41:15.827566 7080 leader_election.cc:302] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 04babc8b97154b25ac1e359afb91c667, aaa07f27c66948e89a3b73053bcaafc0; no voters: 42be682b019e44ed8f1a42e5cf28499f
I20260927 12:41:15.827749 7800 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 12:41:15.827797 7800 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 42be682b019e44ed8f1a42e5cf28499f)
I20260927 12:41:15.827797 7801 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aaa07f27c66948e89a3b73053bcaafc0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 12:41:15.827836 7800 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667 [term 1 FOLLOWER]: Advancing to term 2
I20260927 12:41:15.827865 7801 raft_consensus.cc:482] T 00000000000000000000000000000000 P aaa07f27c66948e89a3b73053bcaafc0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 42be682b019e44ed8f1a42e5cf28499f)
I20260927 12:41:15.827913 7801 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aaa07f27c66948e89a3b73053bcaafc0 [term 1 FOLLOWER]: Advancing to term 2
I20260927 12:41:15.828785 7800 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "42be682b019e44ed8f1a42e5cf28499f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46303 } } peers { permanent_uuid: "04babc8b97154b25ac1e359afb91c667" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46395 } attrs { promote: false } } peers { permanent_uuid: "aaa07f27c66948e89a3b73053bcaafc0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42401 } attrs { promote: false } }
I20260927 12:41:15.828805 7801 raft_consensus.cc:504] T 00000000000000000000000000000000 P aaa07f27c66948e89a3b73053bcaafc0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "42be682b019e44ed8f1a42e5cf28499f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46303 } } peers { permanent_uuid: "04babc8b97154b25ac1e359afb91c667" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46395 } attrs { promote: false } } peers { permanent_uuid: "aaa07f27c66948e89a3b73053bcaafc0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42401 } attrs { promote: false } }
I20260927 12:41:15.829100 7800 leader_election.cc:288] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667 [CANDIDATE]: Term 2 election: Requested vote from peers 42be682b019e44ed8f1a42e5cf28499f (127.0.98.62:46303), aaa07f27c66948e89a3b73053bcaafc0 (127.0.98.60:42401)
I20260927 12:41:15.829120 7801 leader_election.cc:288] T 00000000000000000000000000000000 P aaa07f27c66948e89a3b73053bcaafc0 [CANDIDATE]: Term 2 election: Requested vote from peers 42be682b019e44ed8f1a42e5cf28499f (127.0.98.62:46303), 04babc8b97154b25ac1e359afb91c667 (127.0.98.61:46395)
I20260927 12:41:15.829427 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:15.829182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46303 (local address 127.0.0.1:45134)
0927 12:41:15.829322 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:15.829370 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260927 12:41:15.829487 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:15.829161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46303 (local address 127.0.0.1:45130)
0927 12:41:15.829364 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:15.829431 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
I20260927 12:41:15.829487 7288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "04babc8b97154b25ac1e359afb91c667" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "aaa07f27c66948e89a3b73053bcaafc0"
I20260927 12:41:15.829545 7105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "aaa07f27c66948e89a3b73053bcaafc0" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "04babc8b97154b25ac1e359afb91c667"
I20260927 12:41:15.829655 7288 raft_consensus.cc:2379] T 00000000000000000000000000000000 P aaa07f27c66948e89a3b73053bcaafc0 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 04babc8b97154b25ac1e359afb91c667 in current term 2: Already voted for candidate aaa07f27c66948e89a3b73053bcaafc0 in this term.
I20260927 12:41:15.829707 7105 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate aaa07f27c66948e89a3b73053bcaafc0 in current term 2: Already voted for candidate 04babc8b97154b25ac1e359afb91c667 in this term.
I20260927 12:41:15.830176 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:15.829971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46303 (local address 127.0.0.1:45154)
0927 12:41:15.830099 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:15.830134 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260927 12:41:15.830194 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:15.829932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46303 (local address 127.0.0.1:45140)
0927 12:41:15.830075 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:15.830135 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
W20260927 12:41:15.830271 7082 leader_election.cc:334] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 42be682b019e44ed8f1a42e5cf28499f (127.0.98.62:46303): Network error: Client connection negotiation failed: client connection to 127.0.98.62:46303: connect: Connection refused (error 111)
W20260927 12:41:15.830291 7267 leader_election.cc:334] T 00000000000000000000000000000000 P aaa07f27c66948e89a3b73053bcaafc0 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 42be682b019e44ed8f1a42e5cf28499f (127.0.98.62:46303): Network error: Client connection negotiation failed: client connection to 127.0.98.62:46303: connect: Connection refused (error 111)
I20260927 12:41:15.830348 7267 leader_election.cc:302] T 00000000000000000000000000000000 P aaa07f27c66948e89a3b73053bcaafc0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aaa07f27c66948e89a3b73053bcaafc0; no voters: 04babc8b97154b25ac1e359afb91c667, 42be682b019e44ed8f1a42e5cf28499f
I20260927 12:41:15.830382 7082 leader_election.cc:302] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 04babc8b97154b25ac1e359afb91c667; no voters: 42be682b019e44ed8f1a42e5cf28499f, aaa07f27c66948e89a3b73053bcaafc0
I20260927 12:41:15.830471 7801 raft_consensus.cc:2726] T 00000000000000000000000000000000 P aaa07f27c66948e89a3b73053bcaafc0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 12:41:15.830494 7800 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260927 12:41:16.309687 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:16.309022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46303 (local address 127.0.98.2:38843)
0927 12:41:16.309449 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:16.309538 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":96,"threads_started":1}
I20260927 12:41:16.310222 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:16.309136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46303 (local address 127.0.98.1:39225)
0927 12:41:16.310066 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:16.310129 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":854,"thread_start_us":82,"threads_started":1}
I20260927 12:41:16.310774 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:16.310382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46303 (local address 127.0.98.3:53857)
0927 12:41:16.310625 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:16.310681 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":69,"threads_started":1}
I20260927 12:41:17.311811 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:17.310871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46303 (local address 127.0.98.2:42579)
0927 12:41:17.311613 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:17.311680 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":656,"thread_start_us":72,"threads_started":1}
I20260927 12:41:17.312270 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:17.311535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46303 (local address 127.0.98.3:51251)
0927 12:41:17.312099 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:17.312155 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":55,"threads_started":1}
I20260927 12:41:17.312768 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:17.310911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46303 (local address 127.0.98.1:53825)
0927 12:41:17.312569 (+ 1658us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:17.312646 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":935,"thread_start_us":79,"threads_started":1}
I20260927 12:41:17.897831 7816 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:41:17.897964 7816 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "42be682b019e44ed8f1a42e5cf28499f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46303 } } peers { permanent_uuid: "04babc8b97154b25ac1e359afb91c667" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46395 } attrs { promote: false } } peers { permanent_uuid: "aaa07f27c66948e89a3b73053bcaafc0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42401 } attrs { promote: false } }
I20260927 12:41:17.898422 7816 leader_election.cc:288] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 42be682b019e44ed8f1a42e5cf28499f (127.0.98.62:46303), aaa07f27c66948e89a3b73053bcaafc0 (127.0.98.60:42401)
I20260927 12:41:17.898872 7288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "04babc8b97154b25ac1e359afb91c667" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "aaa07f27c66948e89a3b73053bcaafc0" is_pre_election: true
I20260927 12:41:17.898962 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:17.898614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46303 (local address 127.0.0.1:45446)
0927 12:41:17.898826 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:17.898880 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":72,"threads_started":1}
I20260927 12:41:17.899061 7288 raft_consensus.cc:2454] T 00000000000000000000000000000000 P aaa07f27c66948e89a3b73053bcaafc0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04babc8b97154b25ac1e359afb91c667 in term 2.
I20260927 12:41:17.899389 7080 leader_election.cc:302] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04babc8b97154b25ac1e359afb91c667, aaa07f27c66948e89a3b73053bcaafc0; no voters:
I20260927 12:41:17.899608 7816 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260927 12:41:17.899678 7816 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:41:17.899732 7816 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667 [term 2 FOLLOWER]: Advancing to term 3
I20260927 12:41:17.899799 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:17.899574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46303 (local address 127.0.0.1:45456)
0927 12:41:17.899679 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:17.899733 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 12:41:17.899910 7082 leader_election.cc:334] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 42be682b019e44ed8f1a42e5cf28499f (127.0.98.62:46303): Network error: Client connection negotiation failed: client connection to 127.0.98.62:46303: connect: Connection refused (error 111)
I20260927 12:41:17.900764 7816 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "42be682b019e44ed8f1a42e5cf28499f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46303 } } peers { permanent_uuid: "04babc8b97154b25ac1e359afb91c667" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46395 } attrs { promote: false } } peers { permanent_uuid: "aaa07f27c66948e89a3b73053bcaafc0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42401 } attrs { promote: false } }
I20260927 12:41:17.901084 7816 leader_election.cc:288] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667 [CANDIDATE]: Term 3 election: Requested vote from peers 42be682b019e44ed8f1a42e5cf28499f (127.0.98.62:46303), aaa07f27c66948e89a3b73053bcaafc0 (127.0.98.60:42401)
I20260927 12:41:17.901477 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:17.901278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46303 (local address 127.0.0.1:45466)
0927 12:41:17.901371 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:17.901424 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 12:41:17.901470 7288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "04babc8b97154b25ac1e359afb91c667" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "aaa07f27c66948e89a3b73053bcaafc0"
I20260927 12:41:17.901619 7288 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aaa07f27c66948e89a3b73053bcaafc0 [term 2 FOLLOWER]: Advancing to term 3
I20260927 12:41:17.902200 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:17.902017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46303 (local address 127.0.0.1:45472)
0927 12:41:17.902112 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:17.902154 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 12:41:17.902530 7082 leader_election.cc:334] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 42be682b019e44ed8f1a42e5cf28499f (127.0.98.62:46303): Network error: Client connection negotiation failed: client connection to 127.0.98.62:46303: connect: Connection refused (error 111)
I20260927 12:41:17.902786 7288 raft_consensus.cc:2454] T 00000000000000000000000000000000 P aaa07f27c66948e89a3b73053bcaafc0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04babc8b97154b25ac1e359afb91c667 in term 3.
I20260927 12:41:17.903043 7080 leader_election.cc:302] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 04babc8b97154b25ac1e359afb91c667, aaa07f27c66948e89a3b73053bcaafc0; no voters: 42be682b019e44ed8f1a42e5cf28499f
I20260927 12:41:17.903247 7816 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 12:41:17.903419 7816 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667 [term 3 LEADER]: Becoming Leader. State: Replica: 04babc8b97154b25ac1e359afb91c667, State: Running, Role: LEADER
I20260927 12:41:17.903694 7816 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "42be682b019e44ed8f1a42e5cf28499f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46303 } } peers { permanent_uuid: "04babc8b97154b25ac1e359afb91c667" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46395 } attrs { promote: false } } peers { permanent_uuid: "aaa07f27c66948e89a3b73053bcaafc0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42401 } attrs { promote: false } }
I20260927 12:41:17.904992 7819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04babc8b97154b25ac1e359afb91c667. Latest consensus state: current_term: 3 leader_uuid: "04babc8b97154b25ac1e359afb91c667" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "42be682b019e44ed8f1a42e5cf28499f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46303 } } peers { permanent_uuid: "04babc8b97154b25ac1e359afb91c667" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46395 } attrs { promote: false } } peers { permanent_uuid: "aaa07f27c66948e89a3b73053bcaafc0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42401 } attrs { promote: false } } }
I20260927 12:41:17.905108 7819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667 [sys.catalog]: This master's current role is: LEADER
I20260927 12:41:17.905409 7821 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:41:17.907681 7821 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:41:17.908077 7821 catalog_manager.cc:1259] Loaded cluster ID: d7aa23e71b254e6d8a29824dd7eff1f3
I20260927 12:41:17.908119 7821 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:41:17.908602 7821 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:41:17.908931 7821 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667: Loaded TSK: 0
I20260927 12:41:17.909548 7821 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:41:17.976821 7094 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55886:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 12:41:17.979938 7094 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:41:17.983000 7288 raft_consensus.cc:1261] T 00000000000000000000000000000000 P aaa07f27c66948e89a3b73053bcaafc0 [term 3 FOLLOWER]: Refusing update from remote peer 04babc8b97154b25ac1e359afb91c667: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260927 12:41:17.983187 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:17.982701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46303 (local address 127.0.0.1:45484)
0927 12:41:17.982819 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:17.982875 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 12:41:17.983441 7816 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaa07f27c66948e89a3b73053bcaafc0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42401 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260927 12:41:17.983881 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:17.983682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46303 (local address 127.0.0.1:45496)
0927 12:41:17.983782 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:17.983834 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 12:41:17.984081 7082 consensus_peers.cc:487] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667 -> Peer 42be682b019e44ed8f1a42e5cf28499f (127.0.98.62:46303): Couldn't send request to peer 42be682b019e44ed8f1a42e5cf28499f. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46303: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 12:41:17.984933 7823 sys_catalog.cc:455] T 00000000000000000000000000000000 P aaa07f27c66948e89a3b73053bcaafc0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04babc8b97154b25ac1e359afb91c667. Latest consensus state: current_term: 3 leader_uuid: "04babc8b97154b25ac1e359afb91c667" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "42be682b019e44ed8f1a42e5cf28499f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46303 } } peers { permanent_uuid: "04babc8b97154b25ac1e359afb91c667" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46395 } attrs { promote: false } } peers { permanent_uuid: "aaa07f27c66948e89a3b73053bcaafc0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42401 } attrs { promote: false } } }
I20260927 12:41:17.985037 7823 sys_catalog.cc:458] T 00000000000000000000000000000000 P aaa07f27c66948e89a3b73053bcaafc0 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 12:41:17.986083 7816 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "04babc8b97154b25ac1e359afb91c667" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "42be682b019e44ed8f1a42e5cf28499f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46303 } } peers { permanent_uuid: "04babc8b97154b25ac1e359afb91c667" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46395 } attrs { promote: false } } peers { permanent_uuid: "aaa07f27c66948e89a3b73053bcaafc0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42401 } attrs { promote: false } } }
I20260927 12:41:17.986189 7816 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667 [sys.catalog]: This master's current role is: LEADER
I20260927 12:41:17.986452 7823 sys_catalog.cc:455] T 00000000000000000000000000000000 P aaa07f27c66948e89a3b73053bcaafc0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "04babc8b97154b25ac1e359afb91c667" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "42be682b019e44ed8f1a42e5cf28499f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46303 } } peers { permanent_uuid: "04babc8b97154b25ac1e359afb91c667" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46395 } attrs { promote: false } } peers { permanent_uuid: "aaa07f27c66948e89a3b73053bcaafc0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42401 } attrs { promote: false } } }
I20260927 12:41:17.986636 7823 sys_catalog.cc:458] T 00000000000000000000000000000000 P aaa07f27c66948e89a3b73053bcaafc0 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 12:41:17.996841 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:17.991787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43043 (local address 127.0.0.1:37340)
0927 12:41:17.991957 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:17.991972 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:17.992060 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:17.993144 (+ 1084us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:17.993146 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:17.993157 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:41:17.993249 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:17.993258 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:17.994372 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:17.994375 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:17.995751 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:17.995760 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:17.996407 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:17.996411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:17.996509 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:17.996531 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 12:41:17.996602 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 12:41:17.996672 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81,"mutex_wait_us":99}
I20260927 12:41:17.996968 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:17.992290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40415 (local address 127.0.0.1:41046)
0927 12:41:17.992514 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:17.992527 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:17.992610 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:17.993407 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:17.993411 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:17.993425 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:41:17.993541 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:17.993552 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:17.994329 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:17.994335 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:17.995565 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:17.995573 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:17.996533 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:17.996538 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:17.996633 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:17.996654 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 12:41:17.996729 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 12:41:17.996794 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":52,"thread_start_us":99,"threads_started":1}
I20260927 12:41:17.997066 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:17.992119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36220 (local address 127.0.98.1:42501)
0927 12:41:17.992353 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:17.992359 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:17.992567 (+ 208us) server_negotiation.cc:407] Connection header received
0927 12:41:17.992799 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:17.992803 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:17.992859 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:17.992944 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:41:17.993369 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:17.994239 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:17.995867 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:17.996492 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:17.996591 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:17.996649 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:17.996812 (+ 163us) server_negotiation.cc:299] Negotiation successful
0927 12:41:17.996910 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":85,"threads_started":1}
I20260927 12:41:17.997157 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:17.992834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41046 (local address 127.0.98.3:40415)
0927 12:41:17.993185 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:17.993189 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:17.993204 (+ 15us) server_negotiation.cc:407] Connection header received
0927 12:41:17.993252 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:17.993257 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:17.993300 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:17.993400 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:41:17.993674 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:17.994218 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:17.995689 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:17.996300 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:17.996397 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:17.996462 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:17.996746 (+ 284us) server_negotiation.cc:299] Negotiation successful
0927 12:41:17.996824 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":64,"threads_started":1}
I20260927 12:41:17.997416 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:17.991982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42501 (local address 127.0.0.1:36220)
0927 12:41:17.992467 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:17.992483 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:17.992641 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:17.993064 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:17.993069 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:17.993090 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:41:17.993223 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:17.993229 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:17.994380 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:17.994383 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:17.995751 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:17.995760 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:17.996590 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:17.996596 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:17.996678 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:17.996695 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 12:41:17.997236 (+ 541us) client_negotiation.cc:240] Negotiation successful
0927 12:41:17.997306 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"mutex_wait_us":86,"thread_start_us":82,"threads_started":1}
I20260927 12:41:17.997740 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:17.992236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37340 (local address 127.0.98.2:43043)
0927 12:41:17.992912 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:17.992917 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:17.992932 (+ 15us) server_negotiation.cc:407] Connection header received
0927 12:41:17.992981 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:17.992984 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:17.993029 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:17.993341 (+ 312us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:41:17.993451 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:17.994240 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:17.995883 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:17.996292 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:17.996408 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:17.996470 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:17.997496 (+ 1026us) server_negotiation.cc:299] Negotiation successful
0927 12:41:17.997607 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":68,"threads_started":1}
I20260927 12:41:17.997699 7667 tablet_service.cc:1467] Processing CreateTablet for tablet 32fe393eebbe4097acfaa568d69901fe (DEFAULT_TABLE table=test-table [id=66f9c9891f8643a38d28bcb83facd133]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:41:17.997973 7667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32fe393eebbe4097acfaa568d69901fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:41:17.998025 7393 tablet_service.cc:1467] Processing CreateTablet for tablet 32fe393eebbe4097acfaa568d69901fe (DEFAULT_TABLE table=test-table [id=66f9c9891f8643a38d28bcb83facd133]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:41:17.998227 7393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32fe393eebbe4097acfaa568d69901fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:41:18.000164 7834 tablet_bootstrap.cc:492] T 32fe393eebbe4097acfaa568d69901fe P 4ac28926eacd419bad1e9f8b5e063a36: Bootstrap starting.
I20260927 12:41:18.000216 7835 tablet_bootstrap.cc:492] T 32fe393eebbe4097acfaa568d69901fe P 849bb514fb834ec38a1f428e351782d0: Bootstrap starting.
I20260927 12:41:18.000522 7530 tablet_service.cc:1467] Processing CreateTablet for tablet 32fe393eebbe4097acfaa568d69901fe (DEFAULT_TABLE table=test-table [id=66f9c9891f8643a38d28bcb83facd133]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:41:18.000804 7530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32fe393eebbe4097acfaa568d69901fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:41:18.001134 7835 tablet_bootstrap.cc:654] T 32fe393eebbe4097acfaa568d69901fe P 849bb514fb834ec38a1f428e351782d0: Neither blocks nor log segments found. Creating new log.
I20260927 12:41:18.002349 7834 tablet_bootstrap.cc:654] T 32fe393eebbe4097acfaa568d69901fe P 4ac28926eacd419bad1e9f8b5e063a36: Neither blocks nor log segments found. Creating new log.
I20260927 12:41:18.002398 7835 tablet_bootstrap.cc:492] T 32fe393eebbe4097acfaa568d69901fe P 849bb514fb834ec38a1f428e351782d0: No bootstrap required, opened a new log
I20260927 12:41:18.002470 7835 ts_tablet_manager.cc:1397] T 32fe393eebbe4097acfaa568d69901fe P 849bb514fb834ec38a1f428e351782d0: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 12:41:18.002942 7835 raft_consensus.cc:348] T 32fe393eebbe4097acfaa568d69901fe P 849bb514fb834ec38a1f428e351782d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fd3b55ca4154ec69ee88b69c88fb053" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43043 } } peers { permanent_uuid: "4ac28926eacd419bad1e9f8b5e063a36" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42501 } } peers { permanent_uuid: "849bb514fb834ec38a1f428e351782d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40415 } }
I20260927 12:41:18.003026 7837 tablet_bootstrap.cc:492] T 32fe393eebbe4097acfaa568d69901fe P 9fd3b55ca4154ec69ee88b69c88fb053: Bootstrap starting.
I20260927 12:41:18.003103 7835 raft_consensus.cc:374] T 32fe393eebbe4097acfaa568d69901fe P 849bb514fb834ec38a1f428e351782d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:41:18.003158 7835 raft_consensus.cc:729] T 32fe393eebbe4097acfaa568d69901fe P 849bb514fb834ec38a1f428e351782d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 849bb514fb834ec38a1f428e351782d0, State: Initialized, Role: FOLLOWER
I20260927 12:41:18.003309 7835 consensus_queue.cc:260] T 32fe393eebbe4097acfaa568d69901fe P 849bb514fb834ec38a1f428e351782d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fd3b55ca4154ec69ee88b69c88fb053" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43043 } } peers { permanent_uuid: "4ac28926eacd419bad1e9f8b5e063a36" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42501 } } peers { permanent_uuid: "849bb514fb834ec38a1f428e351782d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40415 } }
I20260927 12:41:18.003660 7835 ts_tablet_manager.cc:1428] T 32fe393eebbe4097acfaa568d69901fe P 849bb514fb834ec38a1f428e351782d0: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260927 12:41:18.004118 7837 tablet_bootstrap.cc:654] T 32fe393eebbe4097acfaa568d69901fe P 9fd3b55ca4154ec69ee88b69c88fb053: Neither blocks nor log segments found. Creating new log.
I20260927 12:41:18.005240 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:18.004390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46303 (local address 127.0.98.3:33037)
0927 12:41:18.004637 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:18.005138 (+ 501us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":98,"threads_started":1}
I20260927 12:41:18.005275 7834 tablet_bootstrap.cc:492] T 32fe393eebbe4097acfaa568d69901fe P 4ac28926eacd419bad1e9f8b5e063a36: No bootstrap required, opened a new log
I20260927 12:41:18.005366 7834 ts_tablet_manager.cc:1397] T 32fe393eebbe4097acfaa568d69901fe P 4ac28926eacd419bad1e9f8b5e063a36: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260927 12:41:18.005946 7834 raft_consensus.cc:348] T 32fe393eebbe4097acfaa568d69901fe P 4ac28926eacd419bad1e9f8b5e063a36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fd3b55ca4154ec69ee88b69c88fb053" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43043 } } peers { permanent_uuid: "4ac28926eacd419bad1e9f8b5e063a36" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42501 } } peers { permanent_uuid: "849bb514fb834ec38a1f428e351782d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40415 } }
I20260927 12:41:18.006057 7834 raft_consensus.cc:374] T 32fe393eebbe4097acfaa568d69901fe P 4ac28926eacd419bad1e9f8b5e063a36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:41:18.006089 7834 raft_consensus.cc:729] T 32fe393eebbe4097acfaa568d69901fe P 4ac28926eacd419bad1e9f8b5e063a36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ac28926eacd419bad1e9f8b5e063a36, State: Initialized, Role: FOLLOWER
I20260927 12:41:18.006246 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:18.005785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46303 (local address 127.0.98.3:56897)
0927 12:41:18.006122 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:18.006177 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":111,"threads_started":1}
I20260927 12:41:18.006214 7834 consensus_queue.cc:260] T 32fe393eebbe4097acfaa568d69901fe P 4ac28926eacd419bad1e9f8b5e063a36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fd3b55ca4154ec69ee88b69c88fb053" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43043 } } peers { permanent_uuid: "4ac28926eacd419bad1e9f8b5e063a36" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42501 } } peers { permanent_uuid: "849bb514fb834ec38a1f428e351782d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40415 } }
I20260927 12:41:18.006577 7834 ts_tablet_manager.cc:1428] T 32fe393eebbe4097acfaa568d69901fe P 4ac28926eacd419bad1e9f8b5e063a36: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 12:41:18.006580 7735 heartbeater.cc:503] Master 127.0.98.61:46395 was elected leader, sending a full tablet report...
I20260927 12:41:18.007346 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:18.006935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46303 (local address 127.0.98.1:54775)
0927 12:41:18.007209 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:18.007263 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":93,"threads_started":1}
I20260927 12:41:18.008476 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:18.008126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46303 (local address 127.0.98.1:32923)
0927 12:41:18.008331 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:18.008382 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":84,"threads_started":1}
I20260927 12:41:18.008469 7837 tablet_bootstrap.cc:492] T 32fe393eebbe4097acfaa568d69901fe P 9fd3b55ca4154ec69ee88b69c88fb053: No bootstrap required, opened a new log
I20260927 12:41:18.008564 7837 ts_tablet_manager.cc:1397] T 32fe393eebbe4097acfaa568d69901fe P 9fd3b55ca4154ec69ee88b69c88fb053: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260927 12:41:18.008985 7461 heartbeater.cc:503] Master 127.0.98.61:46395 was elected leader, sending a full tablet report...
I20260927 12:41:18.008997 7837 raft_consensus.cc:348] T 32fe393eebbe4097acfaa568d69901fe P 9fd3b55ca4154ec69ee88b69c88fb053 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fd3b55ca4154ec69ee88b69c88fb053" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43043 } } peers { permanent_uuid: "4ac28926eacd419bad1e9f8b5e063a36" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42501 } } peers { permanent_uuid: "849bb514fb834ec38a1f428e351782d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40415 } }
I20260927 12:41:18.009099 7837 raft_consensus.cc:374] T 32fe393eebbe4097acfaa568d69901fe P 9fd3b55ca4154ec69ee88b69c88fb053 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:41:18.009141 7837 raft_consensus.cc:729] T 32fe393eebbe4097acfaa568d69901fe P 9fd3b55ca4154ec69ee88b69c88fb053 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fd3b55ca4154ec69ee88b69c88fb053, State: Initialized, Role: FOLLOWER
I20260927 12:41:18.009264 7837 consensus_queue.cc:260] T 32fe393eebbe4097acfaa568d69901fe P 9fd3b55ca4154ec69ee88b69c88fb053 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fd3b55ca4154ec69ee88b69c88fb053" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43043 } } peers { permanent_uuid: "4ac28926eacd419bad1e9f8b5e063a36" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42501 } } peers { permanent_uuid: "849bb514fb834ec38a1f428e351782d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40415 } }
I20260927 12:41:18.009635 7837 ts_tablet_manager.cc:1428] T 32fe393eebbe4097acfaa568d69901fe P 9fd3b55ca4154ec69ee88b69c88fb053: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 12:41:18.010962 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:18.010051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46303 (local address 127.0.98.2:42835)
0927 12:41:18.010765 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:18.010839 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":81,"threads_started":1}
I20260927 12:41:18.011310 7598 heartbeater.cc:503] Master 127.0.98.61:46395 was elected leader, sending a full tablet report...
I20260927 12:41:18.011987 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:18.011519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46303 (local address 127.0.98.2:42287)
0927 12:41:18.011818 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:18.011907 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":100,"threads_started":1}
W20260927 12:41:18.083727 7736 tablet.cc:2369] T 32fe393eebbe4097acfaa568d69901fe P 849bb514fb834ec38a1f428e351782d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 12:41:18.132516 7599 tablet.cc:2369] T 32fe393eebbe4097acfaa568d69901fe P 9fd3b55ca4154ec69ee88b69c88fb053: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:41:18.311287 7846 raft_consensus.cc:482] T 32fe393eebbe4097acfaa568d69901fe P 9fd3b55ca4154ec69ee88b69c88fb053 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:41:18.311443 7846 raft_consensus.cc:504] T 32fe393eebbe4097acfaa568d69901fe P 9fd3b55ca4154ec69ee88b69c88fb053 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fd3b55ca4154ec69ee88b69c88fb053" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43043 } } peers { permanent_uuid: "4ac28926eacd419bad1e9f8b5e063a36" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42501 } } peers { permanent_uuid: "849bb514fb834ec38a1f428e351782d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40415 } }
I20260927 12:41:18.311975 7846 leader_election.cc:288] T 32fe393eebbe4097acfaa568d69901fe P 9fd3b55ca4154ec69ee88b69c88fb053 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ac28926eacd419bad1e9f8b5e063a36 (127.0.98.1:42501), 849bb514fb834ec38a1f428e351782d0 (127.0.98.3:40415)
I20260927 12:41:18.312423 7413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32fe393eebbe4097acfaa568d69901fe" candidate_uuid: "9fd3b55ca4154ec69ee88b69c88fb053" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ac28926eacd419bad1e9f8b5e063a36" is_pre_election: true
I20260927 12:41:18.312600 7413 raft_consensus.cc:2454] T 32fe393eebbe4097acfaa568d69901fe P 4ac28926eacd419bad1e9f8b5e063a36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9fd3b55ca4154ec69ee88b69c88fb053 in term 0.
I20260927 12:41:18.312533 7687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32fe393eebbe4097acfaa568d69901fe" candidate_uuid: "9fd3b55ca4154ec69ee88b69c88fb053" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "849bb514fb834ec38a1f428e351782d0" is_pre_election: true
I20260927 12:41:18.312733 7687 raft_consensus.cc:2454] T 32fe393eebbe4097acfaa568d69901fe P 849bb514fb834ec38a1f428e351782d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9fd3b55ca4154ec69ee88b69c88fb053 in term 0.
I20260927 12:41:18.312980 7486 leader_election.cc:302] T 32fe393eebbe4097acfaa568d69901fe P 9fd3b55ca4154ec69ee88b69c88fb053 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ac28926eacd419bad1e9f8b5e063a36, 9fd3b55ca4154ec69ee88b69c88fb053; no voters:
I20260927 12:41:18.313144 7846 raft_consensus.cc:2781] T 32fe393eebbe4097acfaa568d69901fe P 9fd3b55ca4154ec69ee88b69c88fb053 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:41:18.313222 7846 raft_consensus.cc:482] T 32fe393eebbe4097acfaa568d69901fe P 9fd3b55ca4154ec69ee88b69c88fb053 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:41:18.313268 7846 raft_consensus.cc:3037] T 32fe393eebbe4097acfaa568d69901fe P 9fd3b55ca4154ec69ee88b69c88fb053 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:41:18.314416 7846 raft_consensus.cc:504] T 32fe393eebbe4097acfaa568d69901fe P 9fd3b55ca4154ec69ee88b69c88fb053 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fd3b55ca4154ec69ee88b69c88fb053" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43043 } } peers { permanent_uuid: "4ac28926eacd419bad1e9f8b5e063a36" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42501 } } peers { permanent_uuid: "849bb514fb834ec38a1f428e351782d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40415 } }
I20260927 12:41:18.314744 7846 leader_election.cc:288] T 32fe393eebbe4097acfaa568d69901fe P 9fd3b55ca4154ec69ee88b69c88fb053 [CANDIDATE]: Term 1 election: Requested vote from peers 4ac28926eacd419bad1e9f8b5e063a36 (127.0.98.1:42501), 849bb514fb834ec38a1f428e351782d0 (127.0.98.3:40415)
I20260927 12:41:18.315076 7413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32fe393eebbe4097acfaa568d69901fe" candidate_uuid: "9fd3b55ca4154ec69ee88b69c88fb053" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ac28926eacd419bad1e9f8b5e063a36"
I20260927 12:41:18.315146 7687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32fe393eebbe4097acfaa568d69901fe" candidate_uuid: "9fd3b55ca4154ec69ee88b69c88fb053" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "849bb514fb834ec38a1f428e351782d0"
I20260927 12:41:18.315186 7413 raft_consensus.cc:3037] T 32fe393eebbe4097acfaa568d69901fe P 4ac28926eacd419bad1e9f8b5e063a36 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:41:18.315275 7687 raft_consensus.cc:3037] T 32fe393eebbe4097acfaa568d69901fe P 849bb514fb834ec38a1f428e351782d0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:41:18.316236 7687 raft_consensus.cc:2454] T 32fe393eebbe4097acfaa568d69901fe P 849bb514fb834ec38a1f428e351782d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fd3b55ca4154ec69ee88b69c88fb053 in term 1.
I20260927 12:41:18.316248 7413 raft_consensus.cc:2454] T 32fe393eebbe4097acfaa568d69901fe P 4ac28926eacd419bad1e9f8b5e063a36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fd3b55ca4154ec69ee88b69c88fb053 in term 1.
I20260927 12:41:18.316527 7483 leader_election.cc:302] T 32fe393eebbe4097acfaa568d69901fe P 9fd3b55ca4154ec69ee88b69c88fb053 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 849bb514fb834ec38a1f428e351782d0, 9fd3b55ca4154ec69ee88b69c88fb053; no voters:
I20260927 12:41:18.316726 7846 raft_consensus.cc:2781] T 32fe393eebbe4097acfaa568d69901fe P 9fd3b55ca4154ec69ee88b69c88fb053 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:41:18.316978 7846 raft_consensus.cc:686] T 32fe393eebbe4097acfaa568d69901fe P 9fd3b55ca4154ec69ee88b69c88fb053 [term 1 LEADER]: Becoming Leader. State: Replica: 9fd3b55ca4154ec69ee88b69c88fb053, State: Running, Role: LEADER
I20260927 12:41:18.317171 7846 consensus_queue.cc:237] T 32fe393eebbe4097acfaa568d69901fe P 9fd3b55ca4154ec69ee88b69c88fb053 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fd3b55ca4154ec69ee88b69c88fb053" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43043 } } peers { permanent_uuid: "4ac28926eacd419bad1e9f8b5e063a36" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42501 } } peers { permanent_uuid: "849bb514fb834ec38a1f428e351782d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40415 } }
I20260927 12:41:18.317834 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:18.317447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46303 (local address 127.0.98.2:39981)
0927 12:41:18.317656 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:18.317722 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260927 12:41:18.319170 7094 catalog_manager.cc:5495] T 32fe393eebbe4097acfaa568d69901fe P 9fd3b55ca4154ec69ee88b69c88fb053 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9fd3b55ca4154ec69ee88b69c88fb053 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "9fd3b55ca4154ec69ee88b69c88fb053" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9fd3b55ca4154ec69ee88b69c88fb053" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43043 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4ac28926eacd419bad1e9f8b5e063a36" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "849bb514fb834ec38a1f428e351782d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40415 } health_report { overall_health: UNKNOWN } } }
I20260927 12:41:18.358480 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:18.355153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37348 (local address 127.0.98.2:43043)
0927 12:41:18.355256 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:18.355262 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:18.355334 (+ 72us) server_negotiation.cc:407] Connection header received
0927 12:41:18.355503 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:18.355506 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:18.355584 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:18.355668 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:41:18.355911 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:18.356494 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:18.357367 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:18.357646 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:18.357755 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:18.358005 (+ 250us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:41:18.358079 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:41:18.358159 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:18.358253 (+ 94us) server_negotiation.cc:299] Negotiation successful
0927 12:41:18.358317 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 12:41:18.360607 7687 raft_consensus.cc:1261] T 32fe393eebbe4097acfaa568d69901fe P 849bb514fb834ec38a1f428e351782d0 [term 1 FOLLOWER]: Refusing update from remote peer 9fd3b55ca4154ec69ee88b69c88fb053: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:41:18.360651 7413 raft_consensus.cc:1261] T 32fe393eebbe4097acfaa568d69901fe P 4ac28926eacd419bad1e9f8b5e063a36 [term 1 FOLLOWER]: Refusing update from remote peer 9fd3b55ca4154ec69ee88b69c88fb053: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:41:18.361063 7846 consensus_queue.cc:1035] T 32fe393eebbe4097acfaa568d69901fe P 9fd3b55ca4154ec69ee88b69c88fb053 [LEADER]: Connected to new peer: Peer: permanent_uuid: "849bb514fb834ec38a1f428e351782d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:41:18.361290 7849 consensus_queue.cc:1035] T 32fe393eebbe4097acfaa568d69901fe P 9fd3b55ca4154ec69ee88b69c88fb053 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ac28926eacd419bad1e9f8b5e063a36" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:41:18.362746 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:18.362490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46303 (local address 127.0.98.3:53005)
0927 12:41:18.362583 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:18.362648 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260927 12:41:18.364388 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:18.364171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46303 (local address 127.0.98.1:49169)
0927 12:41:18.364261 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:18.364319 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260927 12:41:18.364940 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:18.364726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46303 (local address 127.0.98.2:55303)
0927 12:41:18.364818 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:18.364872 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260927 12:41:18.365993 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:18.365711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46303 (local address 127.0.98.3:43035)
0927 12:41:18.365871 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:18.365926 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260927 12:41:18.366228 7853 mvcc.cc:205] Tried to move back new op lower bound from 7333940749760417792 to 7333940749587988480. Current Snapshot: MvccSnapshot[applied={T|T < 7333940749760417792}]
I20260927 12:41:18.367760 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:18.367552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46303 (local address 127.0.98.2:55583)
0927 12:41:18.367646 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:18.367699 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260927 12:41:18.368690 7855 mvcc.cc:205] Tried to move back new op lower bound from 7333940749760417792 to 7333940749587988480. Current Snapshot: MvccSnapshot[applied={T|T < 7333940749760417792 or (T in {7333940749760417792})}]
I20260927 12:41:18.369053 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:18.368775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46303 (local address 127.0.98.1:51843)
0927 12:41:18.368939 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:18.368988 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46303: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260927 12:41:18.373026 7094 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55886:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:41:18.373162 7094 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55886:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:41:18.377872 7094 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 04babc8b97154b25ac1e359afb91c667: Sending DeleteTablet for 3 replicas of tablet 32fe393eebbe4097acfaa568d69901fe
I20260927 12:41:18.378459 7530 tablet_service.cc:1514] Processing DeleteTablet for tablet 32fe393eebbe4097acfaa568d69901fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:41:18 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:37340
I20260927 12:41:18.378558 7667 tablet_service.cc:1514] Processing DeleteTablet for tablet 32fe393eebbe4097acfaa568d69901fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:41:18 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:41046
I20260927 12:41:18.378558 7393 tablet_service.cc:1514] Processing DeleteTablet for tablet 32fe393eebbe4097acfaa568d69901fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:41:18 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:36220
I20260927 12:41:18.378803 7861 tablet_replica.cc:330] T 32fe393eebbe4097acfaa568d69901fe P 9fd3b55ca4154ec69ee88b69c88fb053: stopping tablet replica
I20260927 12:41:18.378887 7862 tablet_replica.cc:330] T 32fe393eebbe4097acfaa568d69901fe P 4ac28926eacd419bad1e9f8b5e063a36: stopping tablet replica
I20260927 12:41:18.378955 7861 raft_consensus.cc:2229] T 32fe393eebbe4097acfaa568d69901fe P 9fd3b55ca4154ec69ee88b69c88fb053 [term 1 LEADER]: Raft consensus shutting down.
I20260927 12:41:18.379007 7862 raft_consensus.cc:2229] T 32fe393eebbe4097acfaa568d69901fe P 4ac28926eacd419bad1e9f8b5e063a36 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:41:18.379129 7862 raft_consensus.cc:2258] T 32fe393eebbe4097acfaa568d69901fe P 4ac28926eacd419bad1e9f8b5e063a36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:41:18.379360 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 7240
I20260927 12:41:18.379232 7861 raft_consensus.cc:2258] T 32fe393eebbe4097acfaa568d69901fe P 9fd3b55ca4154ec69ee88b69c88fb053 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:41:18.380728 7863 tablet_replica.cc:330] T 32fe393eebbe4097acfaa568d69901fe P 849bb514fb834ec38a1f428e351782d0: stopping tablet replica
I20260927 12:41:18.380831 7863 raft_consensus.cc:2229] T 32fe393eebbe4097acfaa568d69901fe P 849bb514fb834ec38a1f428e351782d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:41:18.380831 7861 ts_tablet_manager.cc:1905] T 32fe393eebbe4097acfaa568d69901fe P 9fd3b55ca4154ec69ee88b69c88fb053: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:41:18.380920 7863 raft_consensus.cc:2258] T 32fe393eebbe4097acfaa568d69901fe P 849bb514fb834ec38a1f428e351782d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:41:18.381464 7863 ts_tablet_manager.cc:1905] T 32fe393eebbe4097acfaa568d69901fe P 849bb514fb834ec38a1f428e351782d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:41:18.383602 7863 ts_tablet_manager.cc:1918] T 32fe393eebbe4097acfaa568d69901fe P 849bb514fb834ec38a1f428e351782d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:41:18.383724 7863 log.cc:1196] T 32fe393eebbe4097acfaa568d69901fe P 849bb514fb834ec38a1f428e351782d0: Deleting WAL directory at /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-2/wal/wals/32fe393eebbe4097acfaa568d69901fe
I20260927 12:41:18.384032 7863 ts_tablet_manager.cc:1939] T 32fe393eebbe4097acfaa568d69901fe P 849bb514fb834ec38a1f428e351782d0: Deleting consensus metadata
I20260927 12:41:18.384164 7861 ts_tablet_manager.cc:1918] T 32fe393eebbe4097acfaa568d69901fe P 9fd3b55ca4154ec69ee88b69c88fb053: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:41:18.384249 7861 log.cc:1196] T 32fe393eebbe4097acfaa568d69901fe P 9fd3b55ca4154ec69ee88b69c88fb053: Deleting WAL directory at /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790512840836255-392-0/minicluster-data/ts-1/wal/wals/32fe393eebbe4097acfaa568d69901fe
I20260927 12:41:18.384588 7861 ts_tablet_manager.cc:1939] T 32fe393eebbe4097acfaa568d69901fe P 9fd3b55ca4154ec69ee88b69c88fb053: Deleting consensus metadata
I20260927 12:41:18.384702 7079 catalog_manager.cc:4841] TS 849bb514fb834ec38a1f428e351782d0 (127.0.98.3:40415): tablet 32fe393eebbe4097acfaa568d69901fe (table test-table [id=66f9c9891f8643a38d28bcb83facd133]) successfully deleted
I20260927 12:41:18.385250 7079 catalog_manager.cc:4841] TS 9fd3b55ca4154ec69ee88b69c88fb053 (127.0.98.2:43043): tablet 32fe393eebbe4097acfaa568d69901fe (table test-table [id=66f9c9891f8643a38d28bcb83facd133]) successfully deleted
I20260927 12:41:18.387643 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 7467
I20260927 12:41:18.388440 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:18.387804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42501 (local address 127.0.0.1:36226)
0927 12:41:18.388288 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:18.388348 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391}
W20260927 12:41:18.388814 7082 catalog_manager.cc:4584] TS 4ac28926eacd419bad1e9f8b5e063a36 (127.0.98.1:42501): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 32fe393eebbe4097acfaa568d69901fe: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42501: connect: Connection refused (error 111)
I20260927 12:41:18.394745 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 7603
I20260927 12:41:18.400170 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 6959
I20260927 12:41:18.405715 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 7061
2026-09-27T12:41:18Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7668 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:41:18 dist-test-slave-bnc1 krb5kdc[7866](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:41:18 dist-test-slave-bnc1 krb5kdc[7866](info): set up 2 sockets
Sep 27 12:41:18 dist-test-slave-bnc1 krb5kdc[7866](info): commencing operation
krb5kdc: starting...
W20260927 12:41:19.446323 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/krb5kdc/test-user.keytab.
Sep 27 12:41:19 dist-test-slave-bnc1 krb5kdc[7866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512879, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:41:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:41:19Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260927 12:41:19.568351 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41027
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41027
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:41:19.677157 7879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:41:19.677459 7879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:41:19.677508 7879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:41:19.681612 7879 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:41:19.681674 7879 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 12:41:19.681691 7879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:41:19.681710 7879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:41:19.681725 7879 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:41:19.681740 7879 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:41:19.687131 7879 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41027
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7879
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:41:19.688186 7879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:41:19.689177 7879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:41:19.697888 7887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:19.697959 7879 server_base.cc:1034] running on GCE node
W20260927 12:41:19.697908 7885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:19.697893 7884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:19.698750 7879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:41:19.699839 7879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:41:19.701038 7879 hybrid_clock.cc:648] HybridClock initialized: now 1790512879701029 us; error 30 us; skew 500 ppm
Sep 27 12:41:19 dist-test-slave-bnc1 krb5kdc[7866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512879, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:41:19.703771 7879 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260927 12:41:19.704649 7879 webserver.cc:460] Webserver started at http://127.0.98.62:34903/ using document root <none> and password file <none>
I20260927 12:41:19.705215 7879 fs_manager.cc:359] Metadata directory not provided
I20260927 12:41:19.705307 7879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:41:19.705538 7879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:41:19.707583 7879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/minicluster-data/master-0/data/instance:
uuid: "83143be63ef94d3b802195c365bfe42a"
format_stamp: "Formatted at 2026-09-27 12:41:19 on dist-test-slave-bnc1"
server_key: "aa28f6f7df887045a1a1b791084b90af"
server_key_iv: "c2f775960915292bf4492d1b6a2e22a7"
server_key_version: "encryptionkey@0"
I20260927 12:41:19.708135 7879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/minicluster-data/master-0/wal/instance:
uuid: "83143be63ef94d3b802195c365bfe42a"
format_stamp: "Formatted at 2026-09-27 12:41:19 on dist-test-slave-bnc1"
server_key: "aa28f6f7df887045a1a1b791084b90af"
server_key_iv: "c2f775960915292bf4492d1b6a2e22a7"
server_key_version: "encryptionkey@0"
I20260927 12:41:19.711690 7879 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.002s
I20260927 12:41:19.713989 7894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:19.715334 7879 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 12:41:19.715497 7879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/minicluster-data/master-0/wal
uuid: "83143be63ef94d3b802195c365bfe42a"
format_stamp: "Formatted at 2026-09-27 12:41:19 on dist-test-slave-bnc1"
server_key: "aa28f6f7df887045a1a1b791084b90af"
server_key_iv: "c2f775960915292bf4492d1b6a2e22a7"
server_key_version: "encryptionkey@0"
I20260927 12:41:19.715611 7879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:41:19.732662 7879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:41:19.735550 7879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:41:19.735747 7879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:41:19.743422 7879 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41027
I20260927 12:41:19.743443 7956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41027 every 8 connection(s)
I20260927 12:41:19.744445 7879 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:41:19.747154 7957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:41:19.752530 7957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83143be63ef94d3b802195c365bfe42a: Bootstrap starting.
I20260927 12:41:19.754376 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 7879
I20260927 12:41:19.754465 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/minicluster-data/master-0/wal/instance
I20260927 12:41:19.754714 392 external_mini_cluster.cc:1428] Setting key 8002dcddf5a25a6f8b8b9dbb2261ba85
I20260927 12:41:19.755036 7957 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83143be63ef94d3b802195c365bfe42a: Neither blocks nor log segments found. Creating new log.
I20260927 12:41:19.755923 7957 log.cc:824] T 00000000000000000000000000000000 P 83143be63ef94d3b802195c365bfe42a: Log is configured to *not* fsync() on all Append() calls
I20260927 12:41:19.758092 7957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83143be63ef94d3b802195c365bfe42a: No bootstrap required, opened a new log
I20260927 12:41:19.762305 7957 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83143be63ef94d3b802195c365bfe42a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83143be63ef94d3b802195c365bfe42a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41027 } }
I20260927 12:41:19.762578 7957 raft_consensus.cc:374] T 00000000000000000000000000000000 P 83143be63ef94d3b802195c365bfe42a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:41:19.762671 7957 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83143be63ef94d3b802195c365bfe42a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83143be63ef94d3b802195c365bfe42a, State: Initialized, Role: FOLLOWER
I20260927 12:41:19.763080 7957 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83143be63ef94d3b802195c365bfe42a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83143be63ef94d3b802195c365bfe42a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41027 } }
I20260927 12:41:19.763192 7957 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83143be63ef94d3b802195c365bfe42a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:41:19.763273 7957 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83143be63ef94d3b802195c365bfe42a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:41:19.763365 7957 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83143be63ef94d3b802195c365bfe42a [term 0 FOLLOWER]: Advancing to term 1
Sep 27 12:41:19 dist-test-slave-bnc1 krb5kdc[7866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512879, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 12:41:19.764470 7957 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83143be63ef94d3b802195c365bfe42a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83143be63ef94d3b802195c365bfe42a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41027 } }
I20260927 12:41:19.764866 7957 leader_election.cc:302] T 00000000000000000000000000000000 P 83143be63ef94d3b802195c365bfe42a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83143be63ef94d3b802195c365bfe42a; no voters:
I20260927 12:41:19.765221 7957 leader_election.cc:288] T 00000000000000000000000000000000 P 83143be63ef94d3b802195c365bfe42a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:41:19.765283 7962 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83143be63ef94d3b802195c365bfe42a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:41:19.765493 7962 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83143be63ef94d3b802195c365bfe42a [term 1 LEADER]: Becoming Leader. State: Replica: 83143be63ef94d3b802195c365bfe42a, State: Running, Role: LEADER
I20260927 12:41:19.765882 7962 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83143be63ef94d3b802195c365bfe42a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83143be63ef94d3b802195c365bfe42a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41027 } }
I20260927 12:41:19.766462 7957 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83143be63ef94d3b802195c365bfe42a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:41:19.767491 7964 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83143be63ef94d3b802195c365bfe42a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "83143be63ef94d3b802195c365bfe42a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83143be63ef94d3b802195c365bfe42a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41027 } } }
I20260927 12:41:19.767606 7964 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83143be63ef94d3b802195c365bfe42a [sys.catalog]: This master's current role is: LEADER
I20260927 12:41:19.768278 7971 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:41:19.768985 7963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83143be63ef94d3b802195c365bfe42a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83143be63ef94d3b802195c365bfe42a. Latest consensus state: current_term: 1 leader_uuid: "83143be63ef94d3b802195c365bfe42a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83143be63ef94d3b802195c365bfe42a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41027 } } }
I20260927 12:41:19.769121 7963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83143be63ef94d3b802195c365bfe42a [sys.catalog]: This master's current role is: LEADER
I20260927 12:41:19.770604 7960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:19.755815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35630 (local address 127.0.98.62:41027)
0927 12:41:19.756274 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:19.756293 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:19.756340 (+ 47us) server_negotiation.cc:407] Connection header received
0927 12:41:19.757138 (+ 798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:19.757161 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:19.757554 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:19.757936 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:19.758671 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:19.760280 (+ 1609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:19.761107 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:19.761853 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:19.762032 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:19.764931 (+ 2899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:19.764936 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:19.764948 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:41:19.765000 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:19.766864 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:19.767475 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:19.767483 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:19.767490 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:19.767622 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:19.767951 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:19.767954 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:19.767956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:19.768440 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:19.768641 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:19.769389 (+ 748us) server_negotiation.cc:299] Negotiation successful
0927 12:41:19.769721 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":147,"threads_started":1}
I20260927 12:41:19.771384 7971 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:41:19.776506 7971 catalog_manager.cc:1347] Generated new cluster ID: bae3516af82a45f0abfa1c69ac15cfac
I20260927 12:41:19.776625 7971 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:41:19.797617 7971 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:41:19.798511 7971 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:41:19.802925 7971 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 83143be63ef94d3b802195c365bfe42a: Generated new TSK 0
I20260927 12:41:19.803664 7971 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260927 12:41:19.839923 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41027
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:41:19.952011 7983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:41:19.952246 7983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:41:19.952289 7983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:41:19.956298 7983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:41:19.956350 7983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:41:19.956432 7983 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 12:41:19.956508 7983 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:41:19.961679 7983 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:41027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:41:19.962671 7983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:41:19.963672 7983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:41:19.973464 7991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:19.973667 7983 server_base.cc:1034] running on GCE node
W20260927 12:41:19.973729 7989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:19.973520 7988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:19.974277 7983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:41:19.975106 7983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:41:19.976352 7983 hybrid_clock.cc:648] HybridClock initialized: now 1790512879976333 us; error 30 us; skew 500 ppm
Sep 27 12:41:19 dist-test-slave-bnc1 krb5kdc[7866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512879, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:41:19.979074 7983 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 12:41:19.979908 7983 webserver.cc:460] Webserver started at http://127.0.98.1:35861/ using document root <none> and password file <none>
I20260927 12:41:19.980450 7983 fs_manager.cc:359] Metadata directory not provided
I20260927 12:41:19.980540 7983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:41:19.980794 7983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:41:19.983019 7983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/minicluster-data/ts-0/data/instance:
uuid: "b75a048aa032489eaa13cba8d059d8ab"
format_stamp: "Formatted at 2026-09-27 12:41:19 on dist-test-slave-bnc1"
server_key: "ab7b682e17b8e57b1467cf2602106944"
server_key_iv: "55e640395e5f3341a357853d829edb87"
server_key_version: "encryptionkey@0"
I20260927 12:41:19.983603 7983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b75a048aa032489eaa13cba8d059d8ab"
format_stamp: "Formatted at 2026-09-27 12:41:19 on dist-test-slave-bnc1"
server_key: "ab7b682e17b8e57b1467cf2602106944"
server_key_iv: "55e640395e5f3341a357853d829edb87"
server_key_version: "encryptionkey@0"
I20260927 12:41:19.987347 7983 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 12:41:19.989876 7998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:19.991079 7983 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 12:41:19.991225 7983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/minicluster-data/ts-0/wal
uuid: "b75a048aa032489eaa13cba8d059d8ab"
format_stamp: "Formatted at 2026-09-27 12:41:19 on dist-test-slave-bnc1"
server_key: "ab7b682e17b8e57b1467cf2602106944"
server_key_iv: "55e640395e5f3341a357853d829edb87"
server_key_version: "encryptionkey@0"
I20260927 12:41:19.991334 7983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:41:20.011835 7983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:41:20.014997 7983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:41:20.015213 7983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:41:20.016613 7983 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:41:20.016688 7983 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:20.016762 7983 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:41:20.016804 7983 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 12:41:20 dist-test-slave-bnc1 krb5kdc[7866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512879, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 12:41:20.027637 7983 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42243
I20260927 12:41:20.027720 8113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42243 every 8 connection(s)
I20260927 12:41:20.028508 7983 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:41:20.031286 7960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:20.017486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35065 (local address 127.0.98.62:41027)
0927 12:41:20.017636 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:20.017640 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:20.018740 (+ 1100us) server_negotiation.cc:407] Connection header received
0927 12:41:20.019813 (+ 1073us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:20.019817 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:20.019875 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:20.019934 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:20.021689 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:20.022382 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:20.024247 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:20.024566 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:20.024648 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:20.027530 (+ 2882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:20.027533 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:20.027535 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:41:20.027557 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:20.029246 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:20.029808 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:20.029811 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:20.029813 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:20.029894 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:20.030237 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:20.030241 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:20.030242 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:20.030417 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:20.030530 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:20.030992 (+ 462us) server_negotiation.cc:299] Negotiation successful
0927 12:41:20.031116 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 12:41:20.032013 8007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:20.018053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41027 (local address 127.0.98.1:35065)
0927 12:41:20.018545 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:20.018586 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:20.019575 (+ 989us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:20.020361 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:20.020370 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:20.020863 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:20.021528 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:20.021548 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:20.022519 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:20.022524 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:20.024095 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:20.024108 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:20.024690 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:20.024696 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:20.024920 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:20.025477 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:20.025498 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:20.027314 (+ 1816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:41:20.029365 (+ 2051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:20.029371 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:20.029384 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:20.029704 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:20.029998 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:20.030001 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:20.030003 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:20.030137 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:20.030529 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:41:20.030535 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:41:20.030764 (+ 229us) client_negotiation.cc:769] Sending connection context
0927 12:41:20.030935 (+ 171us) client_negotiation.cc:240] Negotiation successful
0927 12:41:20.031239 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":119,"threads_started":1}
I20260927 12:41:20.033281 8114 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41027
I20260927 12:41:20.033458 8114 heartbeater.cc:464] Registering TS with master...
I20260927 12:41:20.034060 8114 heartbeater.cc:511] Master 127.0.98.62:41027 requested a full tablet report, sending...
I20260927 12:41:20.035665 7910 ts_manager.cc:194] Registered new tserver with Master: b75a048aa032489eaa13cba8d059d8ab (127.0.98.1:42243)
I20260927 12:41:20.035954 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 7983
I20260927 12:41:20.036084 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/minicluster-data/ts-0/wal/instance
I20260927 12:41:20.036361 392 external_mini_cluster.cc:1428] Setting key 815142043d92cf513e4de50c283a436e
I20260927 12:41:20.036978 7910 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35065
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 12:41:20.045528 7960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:20.035411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51847 (local address 127.0.98.62:41027)
0927 12:41:20.035547 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:20.035550 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:20.037724 (+ 2174us) server_negotiation.cc:407] Connection header received
0927 12:41:20.039037 (+ 1313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:20.039040 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:20.039086 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:20.039168 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:20.039616 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:20.040114 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:20.040910 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:20.041135 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:20.041245 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:20.042362 (+ 1117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:20.042366 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:20.042368 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:41:20.042391 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:20.043826 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:20.044310 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:20.044313 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:20.044315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:20.044393 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:20.044695 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:20.044698 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:20.044700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:20.044871 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:20.044976 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:20.045247 (+ 271us) server_negotiation.cc:299] Negotiation successful
0927 12:41:20.045357 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 12:41:20.045540 8120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:20.035326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41027 (local address 127.0.98.1:51847)
0927 12:41:20.037646 (+ 2320us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:20.037662 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:20.038917 (+ 1255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:20.039203 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:20.039206 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:20.039395 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:20.039500 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:20.039506 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:20.040237 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:20.040242 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:20.040790 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:20.040797 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:20.041248 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:20.041251 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:20.041345 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:20.041895 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:20.041906 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:20.042224 (+ 318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:41:20.043978 (+ 1754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:20.043997 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:20.043999 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:20.044199 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:20.044497 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:20.044500 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:20.044502 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:20.044596 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:20.044965 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:41:20.044968 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:41:20.045071 (+ 103us) client_negotiation.cc:769] Sending connection context
0927 12:41:20.045195 (+ 124us) client_negotiation.cc:240] Negotiation successful
0927 12:41:20.045345 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2218,"thread_start_us":98,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260927 12:41:20.062649 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41027
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:41:20.170158 8123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:41:20.170423 8123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:41:20.170496 8123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:41:20.174624 8123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:41:20.174721 8123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:41:20.174788 8123 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 12:41:20.174871 8123 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:41:20.180482 8123 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:41027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8123
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:41:20.181701 8123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:41:20.182780 8123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:41:20.192778 8131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:20.192796 8128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:20.192792 8129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:20.192982 8123 server_base.cc:1034] running on GCE node
I20260927 12:41:20.193552 8123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:41:20.194389 8123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:41:20.195562 8123 hybrid_clock.cc:648] HybridClock initialized: now 1790512880195540 us; error 35 us; skew 500 ppm
Sep 27 12:41:20 dist-test-slave-bnc1 krb5kdc[7866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512880, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:41:20.198448 8123 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 12:41:20.199357 8123 webserver.cc:460] Webserver started at http://127.0.98.2:39585/ using document root <none> and password file <none>
I20260927 12:41:20.199901 8123 fs_manager.cc:359] Metadata directory not provided
I20260927 12:41:20.200001 8123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:41:20.200254 8123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:41:20.202395 8123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/minicluster-data/ts-1/data/instance:
uuid: "4fba5977b5054d12b8fee23712c1cd79"
format_stamp: "Formatted at 2026-09-27 12:41:20 on dist-test-slave-bnc1"
server_key: "a5e0795664aa754964c06a8fa94ec22a"
server_key_iv: "4049bf2dc85bf394096b58d39021852a"
server_key_version: "encryptionkey@0"
I20260927 12:41:20.202936 8123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4fba5977b5054d12b8fee23712c1cd79"
format_stamp: "Formatted at 2026-09-27 12:41:20 on dist-test-slave-bnc1"
server_key: "a5e0795664aa754964c06a8fa94ec22a"
server_key_iv: "4049bf2dc85bf394096b58d39021852a"
server_key_version: "encryptionkey@0"
I20260927 12:41:20.206616 8123 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260927 12:41:20.209002 8138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:20.210246 8123 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:41:20.210374 8123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/minicluster-data/ts-1/wal
uuid: "4fba5977b5054d12b8fee23712c1cd79"
format_stamp: "Formatted at 2026-09-27 12:41:20 on dist-test-slave-bnc1"
server_key: "a5e0795664aa754964c06a8fa94ec22a"
server_key_iv: "4049bf2dc85bf394096b58d39021852a"
server_key_version: "encryptionkey@0"
I20260927 12:41:20.210479 8123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:41:20.223836 8123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:41:20.226894 8123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:41:20.227116 8123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:41:20.228894 8123 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:41:20.228978 8123 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:20.229054 8123 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:41:20.229108 8123 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 12:41:20 dist-test-slave-bnc1 krb5kdc[7866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512880, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 12:41:20.241009 8123 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41803
I20260927 12:41:20.241041 8253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41803 every 8 connection(s)
I20260927 12:41:20.242292 8123 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:41:20.243011 7960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:20.229466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35477 (local address 127.0.98.62:41027)
0927 12:41:20.229643 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:20.229649 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:20.230500 (+ 851us) server_negotiation.cc:407] Connection header received
0927 12:41:20.231493 (+ 993us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:20.231497 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:20.231560 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:20.231674 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:20.232947 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:20.233635 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:20.235000 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:20.235237 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:20.235351 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:20.238768 (+ 3417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:20.238772 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:20.238774 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:41:20.238806 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:20.240659 (+ 1853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:20.241330 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:20.241332 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:20.241334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:20.241407 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:20.241842 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:20.241844 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:20.241845 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:20.242011 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:20.242091 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:20.242741 (+ 650us) server_negotiation.cc:299] Negotiation successful
0927 12:41:20.242861 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 12:41:20.243870 8147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:20.229898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41027 (local address 127.0.98.2:35477)
0927 12:41:20.230331 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:20.230379 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:20.231219 (+ 840us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:20.231854 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:20.231862 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:20.232299 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:20.232752 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:20.232765 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:20.233798 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:20.233803 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:20.234874 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:20.234885 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:20.235393 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:20.235397 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:20.235593 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:20.236150 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:20.236168 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:20.238594 (+ 2426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:41:20.240832 (+ 2238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:20.240837 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:20.240851 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:20.241220 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:20.241519 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:20.241522 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:20.241524 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:20.241726 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:20.242116 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:41:20.242122 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:41:20.242431 (+ 309us) client_negotiation.cc:769] Sending connection context
0927 12:41:20.242649 (+ 218us) client_negotiation.cc:240] Negotiation successful
0927 12:41:20.242991 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":121,"threads_started":1}
I20260927 12:41:20.245132 8254 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41027
I20260927 12:41:20.245291 8254 heartbeater.cc:464] Registering TS with master...
I20260927 12:41:20.245868 8254 heartbeater.cc:511] Master 127.0.98.62:41027 requested a full tablet report, sending...
I20260927 12:41:20.246907 7911 ts_manager.cc:194] Registered new tserver with Master: 4fba5977b5054d12b8fee23712c1cd79 (127.0.98.2:41803)
I20260927 12:41:20.247648 7911 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35477
I20260927 12:41:20.248600 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 8123
I20260927 12:41:20.248735 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/minicluster-data/ts-1/wal/instance
I20260927 12:41:20.249033 392 external_mini_cluster.cc:1428] Setting key 8fca537c4e805f634eea40a58364e800
I20260927 12:41:20.254899 7960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:20.247310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47011 (local address 127.0.98.62:41027)
0927 12:41:20.247428 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:20.247431 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:20.247638 (+ 207us) server_negotiation.cc:407] Connection header received
0927 12:41:20.247811 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:20.247814 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:20.247857 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:20.247938 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:20.248390 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:20.249008 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:20.249957 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:20.250265 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:20.250375 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:20.251435 (+ 1060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:20.251438 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:20.251441 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:41:20.251463 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:20.253101 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:20.253497 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:20.253503 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:20.253508 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:20.253624 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:20.253899 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:20.253905 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:20.253910 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:20.254213 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:20.254322 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:20.254698 (+ 376us) server_negotiation.cc:299] Negotiation successful
0927 12:41:20.254791 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 12:41:20.255100 8260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:20.247254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41027 (local address 127.0.98.2:47011)
0927 12:41:20.247555 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:20.247566 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:20.247685 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:20.247969 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:20.247971 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:20.248168 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:20.248259 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:20.248264 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:20.249139 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:20.249150 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:20.249844 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:20.249853 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:20.250377 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:20.250381 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:20.250477 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:20.250981 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:20.250992 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:20.251293 (+ 301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:41:20.253210 (+ 1917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:20.253212 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:20.253213 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:20.253401 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:20.253734 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:20.253736 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:20.253737 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:20.253812 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:20.254376 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:41:20.254382 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:41:20.254501 (+ 119us) client_negotiation.cc:769] Sending connection context
0927 12:41:20.254632 (+ 131us) client_negotiation.cc:240] Negotiation successful
0927 12:41:20.254782 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":82,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260927 12:41:20.274627 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41027
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:41:20.385166 8263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:41:20.385429 8263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:41:20.385473 8263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:41:20.389705 8263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:41:20.389765 8263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:41:20.389833 8263 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 12:41:20.389897 8263 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:41:20.395185 8263 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:41027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8263
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:41:20.396250 8263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:41:20.397217 8263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:41:20.406519 8268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:20.406539 8269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:20.406543 8271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:20.406823 8263 server_base.cc:1034] running on GCE node
I20260927 12:41:20.407219 8263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:41:20.407853 8263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:41:20.409013 8263 hybrid_clock.cc:648] HybridClock initialized: now 1790512880408993 us; error 30 us; skew 500 ppm
Sep 27 12:41:20 dist-test-slave-bnc1 krb5kdc[7866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512880, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:41:20.411718 8263 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 12:41:20.412590 8263 webserver.cc:460] Webserver started at http://127.0.98.3:34671/ using document root <none> and password file <none>
I20260927 12:41:20.413096 8263 fs_manager.cc:359] Metadata directory not provided
I20260927 12:41:20.413151 8263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:41:20.413314 8263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:41:20.415434 8263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/minicluster-data/ts-2/data/instance:
uuid: "7a263c43066c4a74b507d658f43b2be3"
format_stamp: "Formatted at 2026-09-27 12:41:20 on dist-test-slave-bnc1"
server_key: "e5f2ba891f35dc4aea353d5087470af9"
server_key_iv: "524149c37dce0fa79648af8bd4da7fcf"
server_key_version: "encryptionkey@0"
I20260927 12:41:20.415961 8263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7a263c43066c4a74b507d658f43b2be3"
format_stamp: "Formatted at 2026-09-27 12:41:20 on dist-test-slave-bnc1"
server_key: "e5f2ba891f35dc4aea353d5087470af9"
server_key_iv: "524149c37dce0fa79648af8bd4da7fcf"
server_key_version: "encryptionkey@0"
I20260927 12:41:20.419819 8263 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 12:41:20.422127 8278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:20.423309 8263 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 12:41:20.423465 8263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/minicluster-data/ts-2/wal
uuid: "7a263c43066c4a74b507d658f43b2be3"
format_stamp: "Formatted at 2026-09-27 12:41:20 on dist-test-slave-bnc1"
server_key: "e5f2ba891f35dc4aea353d5087470af9"
server_key_iv: "524149c37dce0fa79648af8bd4da7fcf"
server_key_version: "encryptionkey@0"
I20260927 12:41:20.423576 8263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:41:20.438763 8263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:41:20.441989 8263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:41:20.442152 8263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:41:20.443612 8263 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:41:20.443698 8263 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:20.443769 8263 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:41:20.443802 8263 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:20.454520 8263 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34955
Sep 27 12:41:20 dist-test-slave-bnc1 krb5kdc[7866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512880, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 12:41:20.454579 8393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34955 every 8 connection(s)
I20260927 12:41:20.455436 8263 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:41:20.458674 7960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:20.444505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47603 (local address 127.0.98.62:41027)
0927 12:41:20.444628 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:20.444632 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:20.445835 (+ 1203us) server_negotiation.cc:407] Connection header received
0927 12:41:20.447035 (+ 1200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:20.447038 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:20.447101 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:20.447217 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:20.448968 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:20.449858 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:20.451736 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:20.451946 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:20.452030 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:20.455207 (+ 3177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:20.455210 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:20.455213 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:41:20.455245 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:20.456865 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:20.457319 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:20.457322 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:20.457323 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:20.457395 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:20.457737 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:20.457739 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:20.457740 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:20.457910 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:20.457989 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:20.458424 (+ 435us) server_negotiation.cc:299] Negotiation successful
0927 12:41:20.458518 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 12:41:20.459427 8287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:20.445099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41027 (local address 127.0.98.3:47603)
0927 12:41:20.445686 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:20.445730 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:20.446791 (+ 1061us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:20.447474 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:20.447487 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:20.447990 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:20.448781 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:20.448806 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:20.450009 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:20.450016 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:20.451601 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:20.451611 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:20.452118 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:20.452123 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:20.452354 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:20.453119 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:20.453146 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:20.455022 (+ 1876us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:41:20.456964 (+ 1942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:20.456969 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:20.456979 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:20.457217 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:20.457468 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:20.457470 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:20.457472 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:20.457643 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:20.457993 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:41:20.457998 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:41:20.458207 (+ 209us) client_negotiation.cc:769] Sending connection context
0927 12:41:20.458362 (+ 155us) client_negotiation.cc:240] Negotiation successful
0927 12:41:20.458598 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":137,"threads_started":1}
I20260927 12:41:20.460453 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 8263
I20260927 12:41:20.460594 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/minicluster-data/ts-2/wal/instance
I20260927 12:41:20.460711 8394 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41027
I20260927 12:41:20.460852 8394 heartbeater.cc:464] Registering TS with master...
I20260927 12:41:20.460922 392 external_mini_cluster.cc:1428] Setting key cfd890a3351ff660c01f177aad6d20d3
I20260927 12:41:20.461413 8394 heartbeater.cc:511] Master 127.0.98.62:41027 requested a full tablet report, sending...
I20260927 12:41:20.462585 7911 ts_manager.cc:194] Registered new tserver with Master: 7a263c43066c4a74b507d658f43b2be3 (127.0.98.3:34955)
I20260927 12:41:20.463320 7911 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47603
I20260927 12:41:20.470640 8400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:20.463010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41027 (local address 127.0.98.3:53357)
0927 12:41:20.463299 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:20.463311 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:20.463437 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:20.463691 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:20.463695 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:20.463923 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:20.464030 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:20.464036 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:20.465066 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:20.465070 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:20.465998 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:20.466007 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:20.466418 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:20.466420 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:20.466485 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:20.467074 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:20.467087 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:20.467515 (+ 428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:41:20.469147 (+ 1632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:20.469150 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:20.469152 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:20.469342 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:20.469715 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:20.469717 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:20.469719 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:20.469803 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:20.470215 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:41:20.470217 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:41:20.470297 (+ 80us) client_negotiation.cc:769] Sending connection context
0927 12:41:20.470377 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 12:41:20.470489 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":98,"threads_started":1}
I20260927 12:41:20.470642 7960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:20.462995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53357 (local address 127.0.98.62:41027)
0927 12:41:20.463120 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:20.463123 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:20.463398 (+ 275us) server_negotiation.cc:407] Connection header received
0927 12:41:20.463519 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:20.463522 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:20.463571 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:20.463655 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:20.464141 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:20.464907 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:20.466111 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:20.466320 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:20.466402 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:20.467653 (+ 1251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:20.467656 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:20.467658 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:41:20.467678 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:20.469043 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:20.469466 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:20.469470 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:20.469472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:20.469608 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:20.469962 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:20.469965 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:20.469966 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:20.470135 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:20.470216 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:20.470438 (+ 222us) server_negotiation.cc:299] Negotiation successful
0927 12:41:20.470533 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 12:41:20.474920 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 12:41:20 dist-test-slave-bnc1 krb5kdc[7866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512880, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 12:41:20 dist-test-slave-bnc1 krb5kdc[7866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:41:20 dist-test-slave-bnc1 krb5kdc[7866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 12:41:20.498368 7960 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:41:20.492207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35644 (local address 127.0.98.62:41027)
0927 12:41:20.492316 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:20.492324 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:20.492339 (+ 15us) server_negotiation.cc:407] Connection header received
0927 12:41:20.492379 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:20.492381 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:20.492418 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:20.492482 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:20.493154 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:20.493752 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:20.494567 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:20.494807 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:20.494895 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:20.498216 (+ 3321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35644: BlockingRecv error: recv got EOF from 127.0.0.1:35644 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 27 12:41:20 dist-test-slave-bnc1 krb5kdc[7866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512880, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 12:41:20.507292 7960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:20.499760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35646 (local address 127.0.98.62:41027)
0927 12:41:20.499877 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:20.499881 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:20.500020 (+ 139us) server_negotiation.cc:407] Connection header received
0927 12:41:20.500100 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:20.500103 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:20.500141 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:20.500204 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:20.500718 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:20.501221 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:20.502019 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:20.502231 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:20.502313 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:20.504534 (+ 2221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:20.504537 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:20.504539 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:41:20.504596 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:20.505931 (+ 1335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:20.506292 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:20.506294 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:20.506296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:20.506367 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:20.506615 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:20.506617 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:20.506618 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:20.506786 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:20.506890 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:20.507072 (+ 182us) server_negotiation.cc:299] Negotiation successful
0927 12:41:20.507161 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 12:41:20.510282 7911 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35646:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 12:41:20.513348 7911 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:41:20.527921 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:20.521771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42243 (local address 127.0.0.1:49476)
0927 12:41:20.522543 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:20.522594 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:20.522735 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:20.523515 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:20.523522 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:20.523543 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:41:20.523695 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:20.523703 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:20.525176 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:20.525179 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:20.526381 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:20.526391 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:20.527539 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:20.527545 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:20.527653 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:20.527672 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 12:41:20.527741 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 12:41:20.527805 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"mutex_wait_us":37,"spinlock_wait_cycles":1792,"thread_start_us":92,"threads_started":1}
I20260927 12:41:20.528227 8418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:20.521898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49476 (local address 127.0.98.1:42243)
0927 12:41:20.522206 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:20.522222 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:20.522876 (+ 654us) server_negotiation.cc:407] Connection header received
0927 12:41:20.522960 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:20.522969 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:20.523190 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:20.523356 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:41:20.523821 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:20.525058 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:20.526476 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:20.527311 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:20.527438 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:20.527609 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:20.527793 (+ 184us) server_negotiation.cc:299] Negotiation successful
0927 12:41:20.527887 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":86,"threads_started":1}
I20260927 12:41:20.527868 8419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:20.521830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34955 (local address 127.0.0.1:45500)
0927 12:41:20.522580 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:20.522594 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:20.522735 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:20.523658 (+ 923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:20.523661 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:20.523681 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:41:20.523861 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:20.523876 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:20.524848 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:20.524853 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:20.526274 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:20.526285 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:20.527279 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:20.527282 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:20.527409 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:20.527450 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 12:41:20.527560 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 12:41:20.527695 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":706,"mutex_wait_us":38,"spinlock_wait_cycles":6144,"thread_start_us":89,"threads_started":1}
I20260927 12:41:20.528470 8421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:20.522534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45500 (local address 127.0.98.3:34955)
0927 12:41:20.523090 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:20.523102 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:20.523123 (+ 21us) server_negotiation.cc:407] Connection header received
0927 12:41:20.523198 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:20.523202 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:20.523400 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:20.523692 (+ 292us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:41:20.523980 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:20.524729 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:20.526385 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:20.527156 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:20.527275 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:20.527461 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:20.528218 (+ 757us) server_negotiation.cc:299] Negotiation successful
0927 12:41:20.528326 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":103,"threads_started":1}
I20260927 12:41:20.528563 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:20.522329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41803 (local address 127.0.0.1:59906)
0927 12:41:20.522944 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:20.522956 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:20.523027 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:20.523681 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:20.523684 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:20.523709 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:41:20.523851 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:20.523859 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:20.525312 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:20.525318 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:20.526471 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:20.526480 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:20.527516 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:20.527520 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:20.527650 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:20.527680 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 12:41:20.527773 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 12:41:20.527847 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"mutex_wait_us":88,"spinlock_wait_cycles":8448,"thread_start_us":96,"threads_started":1}
I20260927 12:41:20.528820 8420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:20.522487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59906 (local address 127.0.98.2:41803)
0927 12:41:20.523270 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:20.523283 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:20.523304 (+ 21us) server_negotiation.cc:407] Connection header received
0927 12:41:20.523370 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:20.523374 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:20.523538 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:20.523979 (+ 441us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:41:20.524104 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:20.525204 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:20.526570 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:20.527388 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:20.527924 (+ 536us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:20.528071 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:20.528119 (+ 48us) server_negotiation.cc:299] Negotiation successful
0927 12:41:20.528201 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":683,"thread_start_us":99,"threads_started":1}
I20260927 12:41:20.531313 8048 tablet_service.cc:1467] Processing CreateTablet for tablet 77567f81021149679cd1afd14ba4accc (DEFAULT_TABLE table=test-table [id=12cfb294043f4c6f8e1a3123ef65bd7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:41:20.531409 8188 tablet_service.cc:1467] Processing CreateTablet for tablet 77567f81021149679cd1afd14ba4accc (DEFAULT_TABLE table=test-table [id=12cfb294043f4c6f8e1a3123ef65bd7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:41:20.531409 8326 tablet_service.cc:1467] Processing CreateTablet for tablet 77567f81021149679cd1afd14ba4accc (DEFAULT_TABLE table=test-table [id=12cfb294043f4c6f8e1a3123ef65bd7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:41:20.532325 8048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77567f81021149679cd1afd14ba4accc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:41:20.532387 8188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77567f81021149679cd1afd14ba4accc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:41:20.532415 8326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77567f81021149679cd1afd14ba4accc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:41:20.538619 8422 tablet_bootstrap.cc:492] T 77567f81021149679cd1afd14ba4accc P b75a048aa032489eaa13cba8d059d8ab: Bootstrap starting.
I20260927 12:41:20.540673 8424 tablet_bootstrap.cc:492] T 77567f81021149679cd1afd14ba4accc P 4fba5977b5054d12b8fee23712c1cd79: Bootstrap starting.
I20260927 12:41:20.540663 8423 tablet_bootstrap.cc:492] T 77567f81021149679cd1afd14ba4accc P 7a263c43066c4a74b507d658f43b2be3: Bootstrap starting.
I20260927 12:41:20.541342 8422 tablet_bootstrap.cc:654] T 77567f81021149679cd1afd14ba4accc P b75a048aa032489eaa13cba8d059d8ab: Neither blocks nor log segments found. Creating new log.
I20260927 12:41:20.542146 8422 log.cc:824] T 77567f81021149679cd1afd14ba4accc P b75a048aa032489eaa13cba8d059d8ab: Log is configured to *not* fsync() on all Append() calls
I20260927 12:41:20.542829 8424 tablet_bootstrap.cc:654] T 77567f81021149679cd1afd14ba4accc P 4fba5977b5054d12b8fee23712c1cd79: Neither blocks nor log segments found. Creating new log.
I20260927 12:41:20.542829 8423 tablet_bootstrap.cc:654] T 77567f81021149679cd1afd14ba4accc P 7a263c43066c4a74b507d658f43b2be3: Neither blocks nor log segments found. Creating new log.
I20260927 12:41:20.543710 8423 log.cc:824] T 77567f81021149679cd1afd14ba4accc P 7a263c43066c4a74b507d658f43b2be3: Log is configured to *not* fsync() on all Append() calls
I20260927 12:41:20.543710 8424 log.cc:824] T 77567f81021149679cd1afd14ba4accc P 4fba5977b5054d12b8fee23712c1cd79: Log is configured to *not* fsync() on all Append() calls
I20260927 12:41:20.544339 8422 tablet_bootstrap.cc:492] T 77567f81021149679cd1afd14ba4accc P b75a048aa032489eaa13cba8d059d8ab: No bootstrap required, opened a new log
I20260927 12:41:20.544553 8422 ts_tablet_manager.cc:1397] T 77567f81021149679cd1afd14ba4accc P b75a048aa032489eaa13cba8d059d8ab: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 12:41:20.546123 8423 tablet_bootstrap.cc:492] T 77567f81021149679cd1afd14ba4accc P 7a263c43066c4a74b507d658f43b2be3: No bootstrap required, opened a new log
I20260927 12:41:20.546121 8424 tablet_bootstrap.cc:492] T 77567f81021149679cd1afd14ba4accc P 4fba5977b5054d12b8fee23712c1cd79: No bootstrap required, opened a new log
I20260927 12:41:20.546316 8423 ts_tablet_manager.cc:1397] T 77567f81021149679cd1afd14ba4accc P 7a263c43066c4a74b507d658f43b2be3: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260927 12:41:20.546316 8424 ts_tablet_manager.cc:1397] T 77567f81021149679cd1afd14ba4accc P 4fba5977b5054d12b8fee23712c1cd79: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260927 12:41:20.547853 8422 raft_consensus.cc:348] T 77567f81021149679cd1afd14ba4accc P b75a048aa032489eaa13cba8d059d8ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75a048aa032489eaa13cba8d059d8ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42243 } } peers { permanent_uuid: "7a263c43066c4a74b507d658f43b2be3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34955 } } peers { permanent_uuid: "4fba5977b5054d12b8fee23712c1cd79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41803 } }
I20260927 12:41:20.548105 8422 raft_consensus.cc:374] T 77567f81021149679cd1afd14ba4accc P b75a048aa032489eaa13cba8d059d8ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:41:20.548178 8422 raft_consensus.cc:729] T 77567f81021149679cd1afd14ba4accc P b75a048aa032489eaa13cba8d059d8ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b75a048aa032489eaa13cba8d059d8ab, State: Initialized, Role: FOLLOWER
I20260927 12:41:20.548626 8422 consensus_queue.cc:260] T 77567f81021149679cd1afd14ba4accc P b75a048aa032489eaa13cba8d059d8ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75a048aa032489eaa13cba8d059d8ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42243 } } peers { permanent_uuid: "7a263c43066c4a74b507d658f43b2be3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34955 } } peers { permanent_uuid: "4fba5977b5054d12b8fee23712c1cd79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41803 } }
I20260927 12:41:20.549243 8114 heartbeater.cc:503] Master 127.0.98.62:41027 was elected leader, sending a full tablet report...
I20260927 12:41:20.549918 8422 ts_tablet_manager.cc:1428] T 77567f81021149679cd1afd14ba4accc P b75a048aa032489eaa13cba8d059d8ab: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 12:41:20.550462 8423 raft_consensus.cc:348] T 77567f81021149679cd1afd14ba4accc P 7a263c43066c4a74b507d658f43b2be3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75a048aa032489eaa13cba8d059d8ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42243 } } peers { permanent_uuid: "7a263c43066c4a74b507d658f43b2be3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34955 } } peers { permanent_uuid: "4fba5977b5054d12b8fee23712c1cd79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41803 } }
I20260927 12:41:20.550719 8423 raft_consensus.cc:374] T 77567f81021149679cd1afd14ba4accc P 7a263c43066c4a74b507d658f43b2be3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:41:20.550829 8423 raft_consensus.cc:729] T 77567f81021149679cd1afd14ba4accc P 7a263c43066c4a74b507d658f43b2be3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a263c43066c4a74b507d658f43b2be3, State: Initialized, Role: FOLLOWER
I20260927 12:41:20.551271 8423 consensus_queue.cc:260] T 77567f81021149679cd1afd14ba4accc P 7a263c43066c4a74b507d658f43b2be3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75a048aa032489eaa13cba8d059d8ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42243 } } peers { permanent_uuid: "7a263c43066c4a74b507d658f43b2be3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34955 } } peers { permanent_uuid: "4fba5977b5054d12b8fee23712c1cd79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41803 } }
I20260927 12:41:20.551903 8394 heartbeater.cc:503] Master 127.0.98.62:41027 was elected leader, sending a full tablet report...
I20260927 12:41:20.552307 8423 ts_tablet_manager.cc:1428] T 77567f81021149679cd1afd14ba4accc P 7a263c43066c4a74b507d658f43b2be3: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 12:41:20.550462 8424 raft_consensus.cc:348] T 77567f81021149679cd1afd14ba4accc P 4fba5977b5054d12b8fee23712c1cd79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75a048aa032489eaa13cba8d059d8ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42243 } } peers { permanent_uuid: "7a263c43066c4a74b507d658f43b2be3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34955 } } peers { permanent_uuid: "4fba5977b5054d12b8fee23712c1cd79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41803 } }
I20260927 12:41:20.552781 8424 raft_consensus.cc:374] T 77567f81021149679cd1afd14ba4accc P 4fba5977b5054d12b8fee23712c1cd79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:41:20.552855 8424 raft_consensus.cc:729] T 77567f81021149679cd1afd14ba4accc P 4fba5977b5054d12b8fee23712c1cd79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fba5977b5054d12b8fee23712c1cd79, State: Initialized, Role: FOLLOWER
I20260927 12:41:20.553275 8424 consensus_queue.cc:260] T 77567f81021149679cd1afd14ba4accc P 4fba5977b5054d12b8fee23712c1cd79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75a048aa032489eaa13cba8d059d8ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42243 } } peers { permanent_uuid: "7a263c43066c4a74b507d658f43b2be3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34955 } } peers { permanent_uuid: "4fba5977b5054d12b8fee23712c1cd79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41803 } }
I20260927 12:41:20.554075 8254 heartbeater.cc:503] Master 127.0.98.62:41027 was elected leader, sending a full tablet report...
I20260927 12:41:20.554299 8424 ts_tablet_manager.cc:1428] T 77567f81021149679cd1afd14ba4accc P 4fba5977b5054d12b8fee23712c1cd79: Time spent starting tablet: real 0.008s user 0.003s sys 0.002s
W20260927 12:41:20.706704 8395 tablet.cc:2369] T 77567f81021149679cd1afd14ba4accc P 7a263c43066c4a74b507d658f43b2be3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 12:41:20.743757 8255 tablet.cc:2369] T 77567f81021149679cd1afd14ba4accc P 4fba5977b5054d12b8fee23712c1cd79: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:41:20.769001 8430 raft_consensus.cc:482] T 77567f81021149679cd1afd14ba4accc P 4fba5977b5054d12b8fee23712c1cd79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:41:20.769268 8430 raft_consensus.cc:504] T 77567f81021149679cd1afd14ba4accc P 4fba5977b5054d12b8fee23712c1cd79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75a048aa032489eaa13cba8d059d8ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42243 } } peers { permanent_uuid: "7a263c43066c4a74b507d658f43b2be3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34955 } } peers { permanent_uuid: "4fba5977b5054d12b8fee23712c1cd79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41803 } }
I20260927 12:41:20.770506 8430 leader_election.cc:288] T 77567f81021149679cd1afd14ba4accc P 4fba5977b5054d12b8fee23712c1cd79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b75a048aa032489eaa13cba8d059d8ab (127.0.98.1:42243), 7a263c43066c4a74b507d658f43b2be3 (127.0.98.3:34955)
I20260927 12:41:20.774417 8421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:20.770873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36291 (local address 127.0.98.3:34955)
0927 12:41:20.771104 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:20.771109 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:20.771270 (+ 161us) server_negotiation.cc:407] Connection header received
0927 12:41:20.771421 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:20.771424 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:20.771472 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:20.771552 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:41:20.771789 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:20.772322 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:20.773365 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:20.773825 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:20.773939 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:20.774004 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:20.774211 (+ 207us) server_negotiation.cc:299] Negotiation successful
0927 12:41:20.774277 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260927 12:41:20.774430 8418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:20.770915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49761 (local address 127.0.98.1:42243)
0927 12:41:20.771067 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:20.771073 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:20.771158 (+ 85us) server_negotiation.cc:407] Connection header received
0927 12:41:20.771317 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:20.771320 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:20.771363 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:20.771446 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:41:20.771719 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:20.772247 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:20.773388 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:20.773852 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:20.773992 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:20.774055 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:20.774238 (+ 183us) server_negotiation.cc:299] Negotiation successful
0927 12:41:20.774304 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 12:41:20.774454 8431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:20.770742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34955 (local address 127.0.98.2:36291)
0927 12:41:20.771190 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:20.771201 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:20.771313 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:20.771561 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:20.771563 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:20.771578 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:41:20.771688 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:20.771693 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:20.772430 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:20.772435 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:20.773251 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:20.773259 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:20.773941 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:20.773945 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:20.774055 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:20.774079 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 12:41:20.774199 (+ 120us) client_negotiation.cc:240] Negotiation successful
0927 12:41:20.774273 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"mutex_wait_us":52,"thread_start_us":73,"threads_started":1}
I20260927 12:41:20.774478 8432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:20.770743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42243 (local address 127.0.98.2:49761)
0927 12:41:20.771063 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:20.771078 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:20.771199 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:20.771456 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:20.771458 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:20.771473 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:41:20.771607 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:20.771612 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:20.772358 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:20.772364 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:20.773266 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:20.773274 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:20.773994 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:20.773998 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:20.774101 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:20.774121 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 12:41:20.774222 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 12:41:20.774297 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":63,"thread_start_us":80,"threads_started":1}
I20260927 12:41:20.775126 8348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77567f81021149679cd1afd14ba4accc" candidate_uuid: "4fba5977b5054d12b8fee23712c1cd79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a263c43066c4a74b507d658f43b2be3" is_pre_election: true
I20260927 12:41:20.775236 8068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77567f81021149679cd1afd14ba4accc" candidate_uuid: "4fba5977b5054d12b8fee23712c1cd79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b75a048aa032489eaa13cba8d059d8ab" is_pre_election: true
I20260927 12:41:20.775475 8348 raft_consensus.cc:2454] T 77567f81021149679cd1afd14ba4accc P 7a263c43066c4a74b507d658f43b2be3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fba5977b5054d12b8fee23712c1cd79 in term 0.
I20260927 12:41:20.775532 8068 raft_consensus.cc:2454] T 77567f81021149679cd1afd14ba4accc P b75a048aa032489eaa13cba8d059d8ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fba5977b5054d12b8fee23712c1cd79 in term 0.
I20260927 12:41:20.776019 8142 leader_election.cc:302] T 77567f81021149679cd1afd14ba4accc P 4fba5977b5054d12b8fee23712c1cd79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4fba5977b5054d12b8fee23712c1cd79, 7a263c43066c4a74b507d658f43b2be3, b75a048aa032489eaa13cba8d059d8ab; no voters:
I20260927 12:41:20.776311 8430 raft_consensus.cc:2781] T 77567f81021149679cd1afd14ba4accc P 4fba5977b5054d12b8fee23712c1cd79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:41:20.776409 8430 raft_consensus.cc:482] T 77567f81021149679cd1afd14ba4accc P 4fba5977b5054d12b8fee23712c1cd79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:41:20.776469 8430 raft_consensus.cc:3037] T 77567f81021149679cd1afd14ba4accc P 4fba5977b5054d12b8fee23712c1cd79 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:41:20.777647 8430 raft_consensus.cc:504] T 77567f81021149679cd1afd14ba4accc P 4fba5977b5054d12b8fee23712c1cd79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75a048aa032489eaa13cba8d059d8ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42243 } } peers { permanent_uuid: "7a263c43066c4a74b507d658f43b2be3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34955 } } peers { permanent_uuid: "4fba5977b5054d12b8fee23712c1cd79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41803 } }
I20260927 12:41:20.778075 8430 leader_election.cc:288] T 77567f81021149679cd1afd14ba4accc P 4fba5977b5054d12b8fee23712c1cd79 [CANDIDATE]: Term 1 election: Requested vote from peers b75a048aa032489eaa13cba8d059d8ab (127.0.98.1:42243), 7a263c43066c4a74b507d658f43b2be3 (127.0.98.3:34955)
I20260927 12:41:20.778466 8348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77567f81021149679cd1afd14ba4accc" candidate_uuid: "4fba5977b5054d12b8fee23712c1cd79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a263c43066c4a74b507d658f43b2be3"
I20260927 12:41:20.778537 8068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77567f81021149679cd1afd14ba4accc" candidate_uuid: "4fba5977b5054d12b8fee23712c1cd79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b75a048aa032489eaa13cba8d059d8ab"
I20260927 12:41:20.778595 8348 raft_consensus.cc:3037] T 77567f81021149679cd1afd14ba4accc P 7a263c43066c4a74b507d658f43b2be3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:41:20.778640 8068 raft_consensus.cc:3037] T 77567f81021149679cd1afd14ba4accc P b75a048aa032489eaa13cba8d059d8ab [term 0 FOLLOWER]: Advancing to term 1
W20260927 12:41:20.779888 8115 tablet.cc:2369] T 77567f81021149679cd1afd14ba4accc P b75a048aa032489eaa13cba8d059d8ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:41:20.779951 8068 raft_consensus.cc:2454] T 77567f81021149679cd1afd14ba4accc P b75a048aa032489eaa13cba8d059d8ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fba5977b5054d12b8fee23712c1cd79 in term 1.
I20260927 12:41:20.780354 8348 raft_consensus.cc:2454] T 77567f81021149679cd1afd14ba4accc P 7a263c43066c4a74b507d658f43b2be3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fba5977b5054d12b8fee23712c1cd79 in term 1.
I20260927 12:41:20.780380 8140 leader_election.cc:302] T 77567f81021149679cd1afd14ba4accc P 4fba5977b5054d12b8fee23712c1cd79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fba5977b5054d12b8fee23712c1cd79, b75a048aa032489eaa13cba8d059d8ab; no voters:
I20260927 12:41:20.780544 8430 raft_consensus.cc:2781] T 77567f81021149679cd1afd14ba4accc P 4fba5977b5054d12b8fee23712c1cd79 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:41:20.780751 8430 raft_consensus.cc:686] T 77567f81021149679cd1afd14ba4accc P 4fba5977b5054d12b8fee23712c1cd79 [term 1 LEADER]: Becoming Leader. State: Replica: 4fba5977b5054d12b8fee23712c1cd79, State: Running, Role: LEADER
I20260927 12:41:20.781081 8430 consensus_queue.cc:237] T 77567f81021149679cd1afd14ba4accc P 4fba5977b5054d12b8fee23712c1cd79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75a048aa032489eaa13cba8d059d8ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42243 } } peers { permanent_uuid: "7a263c43066c4a74b507d658f43b2be3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34955 } } peers { permanent_uuid: "4fba5977b5054d12b8fee23712c1cd79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41803 } }
I20260927 12:41:20.784243 7911 catalog_manager.cc:5495] T 77567f81021149679cd1afd14ba4accc P 4fba5977b5054d12b8fee23712c1cd79 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4fba5977b5054d12b8fee23712c1cd79 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "4fba5977b5054d12b8fee23712c1cd79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75a048aa032489eaa13cba8d059d8ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a263c43066c4a74b507d658f43b2be3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fba5977b5054d12b8fee23712c1cd79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41803 } health_report { overall_health: HEALTHY } } }
I20260927 12:41:20.818439 8420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:20.815300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59916 (local address 127.0.98.2:41803)
0927 12:41:20.815420 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:20.815432 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:20.815445 (+ 13us) server_negotiation.cc:407] Connection header received
0927 12:41:20.815519 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:20.815521 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:20.815573 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:20.815663 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:41:20.815861 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:20.816353 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:20.817128 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:20.817341 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:20.817416 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:20.817659 (+ 243us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:41:20.818073 (+ 414us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:41:20.818156 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:20.818240 (+ 84us) server_negotiation.cc:299] Negotiation successful
0927 12:41:20.818290 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 12:41:20.823974 8348 raft_consensus.cc:1261] T 77567f81021149679cd1afd14ba4accc P 7a263c43066c4a74b507d658f43b2be3 [term 1 FOLLOWER]: Refusing update from remote peer 4fba5977b5054d12b8fee23712c1cd79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:41:20.823993 8068 raft_consensus.cc:1261] T 77567f81021149679cd1afd14ba4accc P b75a048aa032489eaa13cba8d059d8ab [term 1 FOLLOWER]: Refusing update from remote peer 4fba5977b5054d12b8fee23712c1cd79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:41:20.824635 8433 consensus_queue.cc:1035] T 77567f81021149679cd1afd14ba4accc P 4fba5977b5054d12b8fee23712c1cd79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b75a048aa032489eaa13cba8d059d8ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:41:20.824824 8430 consensus_queue.cc:1035] T 77567f81021149679cd1afd14ba4accc P 4fba5977b5054d12b8fee23712c1cd79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a263c43066c4a74b507d658f43b2be3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:41:20.832140 8436 mvcc.cc:205] Tried to move back new op lower bound from 7333940759848488960 to 7333940759682469888. Current Snapshot: MvccSnapshot[applied={T|T < 7333940759848488960}]
I20260927 12:41:20.834293 8438 mvcc.cc:205] Tried to move back new op lower bound from 7333940759848488960 to 7333940759682469888. Current Snapshot: MvccSnapshot[applied={T|T < 7333940759848488960}]
I20260927 12:41:20.840263 7911 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35646:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:41:20.840436 7911 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35646:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:41:20.842677 7911 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 83143be63ef94d3b802195c365bfe42a: Sending DeleteTablet for 3 replicas of tablet 77567f81021149679cd1afd14ba4accc
I20260927 12:41:20.843408 8048 tablet_service.cc:1514] Processing DeleteTablet for tablet 77567f81021149679cd1afd14ba4accc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:41:20 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49476
I20260927 12:41:20.843430 8326 tablet_service.cc:1514] Processing DeleteTablet for tablet 77567f81021149679cd1afd14ba4accc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:41:20 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45500
I20260927 12:41:20.843652 8188 tablet_service.cc:1514] Processing DeleteTablet for tablet 77567f81021149679cd1afd14ba4accc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:41:20 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59906
I20260927 12:41:20.843909 8445 tablet_replica.cc:330] T 77567f81021149679cd1afd14ba4accc P b75a048aa032489eaa13cba8d059d8ab: stopping tablet replica
I20260927 12:41:20.844053 8447 tablet_replica.cc:330] T 77567f81021149679cd1afd14ba4accc P 4fba5977b5054d12b8fee23712c1cd79: stopping tablet replica
I20260927 12:41:20.844153 8446 tablet_replica.cc:330] T 77567f81021149679cd1afd14ba4accc P 7a263c43066c4a74b507d658f43b2be3: stopping tablet replica
I20260927 12:41:20.844376 8447 raft_consensus.cc:2229] T 77567f81021149679cd1afd14ba4accc P 4fba5977b5054d12b8fee23712c1cd79 [term 1 LEADER]: Raft consensus shutting down.
I20260927 12:41:20.844667 8445 raft_consensus.cc:2229] T 77567f81021149679cd1afd14ba4accc P b75a048aa032489eaa13cba8d059d8ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:41:20.844780 8447 raft_consensus.cc:2258] T 77567f81021149679cd1afd14ba4accc P 4fba5977b5054d12b8fee23712c1cd79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:41:20.844707 7911 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35646:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 12:41:20.844817 8446 raft_consensus.cc:2229] T 77567f81021149679cd1afd14ba4accc P 7a263c43066c4a74b507d658f43b2be3 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260927 12:41:20.845122 7911 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:41:20.845259 8445 raft_consensus.cc:2258] T 77567f81021149679cd1afd14ba4accc P b75a048aa032489eaa13cba8d059d8ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:41:20.845719 8446 raft_consensus.cc:2258] T 77567f81021149679cd1afd14ba4accc P 7a263c43066c4a74b507d658f43b2be3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:41:20.846946 8445 ts_tablet_manager.cc:1905] T 77567f81021149679cd1afd14ba4accc P b75a048aa032489eaa13cba8d059d8ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:41:20.847119 8447 ts_tablet_manager.cc:1905] T 77567f81021149679cd1afd14ba4accc P 4fba5977b5054d12b8fee23712c1cd79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:41:20.847425 8446 ts_tablet_manager.cc:1905] T 77567f81021149679cd1afd14ba4accc P 7a263c43066c4a74b507d658f43b2be3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:41:20.850090 8048 tablet_service.cc:1467] Processing CreateTablet for tablet 0d59c48666ac4e14a2eb9c77d9d443bf (DEFAULT_TABLE table=test-table [id=7cc68899e87d4c9db9d7a07f2bf7b51e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:41:20.850258 8326 tablet_service.cc:1467] Processing CreateTablet for tablet 0d59c48666ac4e14a2eb9c77d9d443bf (DEFAULT_TABLE table=test-table [id=7cc68899e87d4c9db9d7a07f2bf7b51e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:41:20.850314 8446 ts_tablet_manager.cc:1918] T 77567f81021149679cd1afd14ba4accc P 7a263c43066c4a74b507d658f43b2be3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:41:20.850399 8446 log.cc:1196] T 77567f81021149679cd1afd14ba4accc P 7a263c43066c4a74b507d658f43b2be3: Deleting WAL directory at /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/minicluster-data/ts-2/wal/wals/77567f81021149679cd1afd14ba4accc
I20260927 12:41:20.850512 8326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d59c48666ac4e14a2eb9c77d9d443bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:41:20.850687 8446 ts_tablet_manager.cc:1939] T 77567f81021149679cd1afd14ba4accc P 7a263c43066c4a74b507d658f43b2be3: Deleting consensus metadata
I20260927 12:41:20.851708 8048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d59c48666ac4e14a2eb9c77d9d443bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:41:20.852619 8423 tablet_bootstrap.cc:492] T 0d59c48666ac4e14a2eb9c77d9d443bf P 7a263c43066c4a74b507d658f43b2be3: Bootstrap starting.
I20260927 12:41:20.852792 7898 catalog_manager.cc:4841] TS 7a263c43066c4a74b507d658f43b2be3 (127.0.98.3:34955): tablet 77567f81021149679cd1afd14ba4accc (table test-table [id=12cfb294043f4c6f8e1a3123ef65bd7f]) successfully deleted
I20260927 12:41:20.853577 8423 tablet_bootstrap.cc:654] T 0d59c48666ac4e14a2eb9c77d9d443bf P 7a263c43066c4a74b507d658f43b2be3: Neither blocks nor log segments found. Creating new log.
I20260927 12:41:20.853907 8422 tablet_bootstrap.cc:492] T 0d59c48666ac4e14a2eb9c77d9d443bf P b75a048aa032489eaa13cba8d059d8ab: Bootstrap starting.
I20260927 12:41:20.854717 8423 tablet_bootstrap.cc:492] T 0d59c48666ac4e14a2eb9c77d9d443bf P 7a263c43066c4a74b507d658f43b2be3: No bootstrap required, opened a new log
I20260927 12:41:20.854807 8423 ts_tablet_manager.cc:1397] T 0d59c48666ac4e14a2eb9c77d9d443bf P 7a263c43066c4a74b507d658f43b2be3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 12:41:20.854848 8422 tablet_bootstrap.cc:654] T 0d59c48666ac4e14a2eb9c77d9d443bf P b75a048aa032489eaa13cba8d059d8ab: Neither blocks nor log segments found. Creating new log.
I20260927 12:41:20.855209 8423 raft_consensus.cc:348] T 0d59c48666ac4e14a2eb9c77d9d443bf P 7a263c43066c4a74b507d658f43b2be3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75a048aa032489eaa13cba8d059d8ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42243 } } peers { permanent_uuid: "4fba5977b5054d12b8fee23712c1cd79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41803 } } peers { permanent_uuid: "7a263c43066c4a74b507d658f43b2be3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34955 } }
I20260927 12:41:20.855319 8423 raft_consensus.cc:374] T 0d59c48666ac4e14a2eb9c77d9d443bf P 7a263c43066c4a74b507d658f43b2be3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:41:20.855350 8423 raft_consensus.cc:729] T 0d59c48666ac4e14a2eb9c77d9d443bf P 7a263c43066c4a74b507d658f43b2be3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a263c43066c4a74b507d658f43b2be3, State: Initialized, Role: FOLLOWER
I20260927 12:41:20.855468 8423 consensus_queue.cc:260] T 0d59c48666ac4e14a2eb9c77d9d443bf P 7a263c43066c4a74b507d658f43b2be3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75a048aa032489eaa13cba8d059d8ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42243 } } peers { permanent_uuid: "4fba5977b5054d12b8fee23712c1cd79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41803 } } peers { permanent_uuid: "7a263c43066c4a74b507d658f43b2be3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34955 } }
I20260927 12:41:20.855710 8423 ts_tablet_manager.cc:1428] T 0d59c48666ac4e14a2eb9c77d9d443bf P 7a263c43066c4a74b507d658f43b2be3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 12:41:20.855868 8445 ts_tablet_manager.cc:1918] T 77567f81021149679cd1afd14ba4accc P b75a048aa032489eaa13cba8d059d8ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:41:20.856320 8445 log.cc:1196] T 77567f81021149679cd1afd14ba4accc P b75a048aa032489eaa13cba8d059d8ab: Deleting WAL directory at /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/minicluster-data/ts-0/wal/wals/77567f81021149679cd1afd14ba4accc
I20260927 12:41:20.857072 8445 ts_tablet_manager.cc:1939] T 77567f81021149679cd1afd14ba4accc P b75a048aa032489eaa13cba8d059d8ab: Deleting consensus metadata
I20260927 12:41:20.850824 8188 tablet_service.cc:1467] Processing CreateTablet for tablet 0d59c48666ac4e14a2eb9c77d9d443bf (DEFAULT_TABLE table=test-table [id=7cc68899e87d4c9db9d7a07f2bf7b51e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:41:20.858304 7896 catalog_manager.cc:4841] TS b75a048aa032489eaa13cba8d059d8ab (127.0.98.1:42243): tablet 77567f81021149679cd1afd14ba4accc (table test-table [id=12cfb294043f4c6f8e1a3123ef65bd7f]) successfully deleted
I20260927 12:41:20.858599 8188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d59c48666ac4e14a2eb9c77d9d443bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:41:20.858875 8422 tablet_bootstrap.cc:492] T 0d59c48666ac4e14a2eb9c77d9d443bf P b75a048aa032489eaa13cba8d059d8ab: No bootstrap required, opened a new log
I20260927 12:41:20.858958 8422 ts_tablet_manager.cc:1397] T 0d59c48666ac4e14a2eb9c77d9d443bf P b75a048aa032489eaa13cba8d059d8ab: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260927 12:41:20.859367 8422 raft_consensus.cc:348] T 0d59c48666ac4e14a2eb9c77d9d443bf P b75a048aa032489eaa13cba8d059d8ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75a048aa032489eaa13cba8d059d8ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42243 } } peers { permanent_uuid: "4fba5977b5054d12b8fee23712c1cd79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41803 } } peers { permanent_uuid: "7a263c43066c4a74b507d658f43b2be3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34955 } }
I20260927 12:41:20.859467 8422 raft_consensus.cc:374] T 0d59c48666ac4e14a2eb9c77d9d443bf P b75a048aa032489eaa13cba8d059d8ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:41:20.859498 8422 raft_consensus.cc:729] T 0d59c48666ac4e14a2eb9c77d9d443bf P b75a048aa032489eaa13cba8d059d8ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b75a048aa032489eaa13cba8d059d8ab, State: Initialized, Role: FOLLOWER
I20260927 12:41:20.859611 8422 consensus_queue.cc:260] T 0d59c48666ac4e14a2eb9c77d9d443bf P b75a048aa032489eaa13cba8d059d8ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75a048aa032489eaa13cba8d059d8ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42243 } } peers { permanent_uuid: "4fba5977b5054d12b8fee23712c1cd79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41803 } } peers { permanent_uuid: "7a263c43066c4a74b507d658f43b2be3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34955 } }
I20260927 12:41:20.859872 8422 ts_tablet_manager.cc:1428] T 0d59c48666ac4e14a2eb9c77d9d443bf P b75a048aa032489eaa13cba8d059d8ab: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 12:41:20.860924 8424 tablet_bootstrap.cc:492] T 0d59c48666ac4e14a2eb9c77d9d443bf P 4fba5977b5054d12b8fee23712c1cd79: Bootstrap starting.
I20260927 12:41:20.861047 8447 ts_tablet_manager.cc:1918] T 77567f81021149679cd1afd14ba4accc P 4fba5977b5054d12b8fee23712c1cd79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:41:20.861120 8447 log.cc:1196] T 77567f81021149679cd1afd14ba4accc P 4fba5977b5054d12b8fee23712c1cd79: Deleting WAL directory at /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/minicluster-data/ts-1/wal/wals/77567f81021149679cd1afd14ba4accc
I20260927 12:41:20.861426 8447 ts_tablet_manager.cc:1939] T 77567f81021149679cd1afd14ba4accc P 4fba5977b5054d12b8fee23712c1cd79: Deleting consensus metadata
I20260927 12:41:20.862003 8424 tablet_bootstrap.cc:654] T 0d59c48666ac4e14a2eb9c77d9d443bf P 4fba5977b5054d12b8fee23712c1cd79: Neither blocks nor log segments found. Creating new log.
I20260927 12:41:20.862841 7897 catalog_manager.cc:4841] TS 4fba5977b5054d12b8fee23712c1cd79 (127.0.98.2:41803): tablet 77567f81021149679cd1afd14ba4accc (table test-table [id=12cfb294043f4c6f8e1a3123ef65bd7f]) successfully deleted
I20260927 12:41:20.863199 8424 tablet_bootstrap.cc:492] T 0d59c48666ac4e14a2eb9c77d9d443bf P 4fba5977b5054d12b8fee23712c1cd79: No bootstrap required, opened a new log
I20260927 12:41:20.863286 8424 ts_tablet_manager.cc:1397] T 0d59c48666ac4e14a2eb9c77d9d443bf P 4fba5977b5054d12b8fee23712c1cd79: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260927 12:41:20.863673 8424 raft_consensus.cc:348] T 0d59c48666ac4e14a2eb9c77d9d443bf P 4fba5977b5054d12b8fee23712c1cd79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75a048aa032489eaa13cba8d059d8ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42243 } } peers { permanent_uuid: "4fba5977b5054d12b8fee23712c1cd79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41803 } } peers { permanent_uuid: "7a263c43066c4a74b507d658f43b2be3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34955 } }
I20260927 12:41:20.863781 8424 raft_consensus.cc:374] T 0d59c48666ac4e14a2eb9c77d9d443bf P 4fba5977b5054d12b8fee23712c1cd79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:41:20.863813 8424 raft_consensus.cc:729] T 0d59c48666ac4e14a2eb9c77d9d443bf P 4fba5977b5054d12b8fee23712c1cd79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fba5977b5054d12b8fee23712c1cd79, State: Initialized, Role: FOLLOWER
I20260927 12:41:20.863924 8424 consensus_queue.cc:260] T 0d59c48666ac4e14a2eb9c77d9d443bf P 4fba5977b5054d12b8fee23712c1cd79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75a048aa032489eaa13cba8d059d8ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42243 } } peers { permanent_uuid: "4fba5977b5054d12b8fee23712c1cd79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41803 } } peers { permanent_uuid: "7a263c43066c4a74b507d658f43b2be3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34955 } }
I20260927 12:41:20.864152 8424 ts_tablet_manager.cc:1428] T 0d59c48666ac4e14a2eb9c77d9d443bf P 4fba5977b5054d12b8fee23712c1cd79: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 12:41:20.897254 8430 raft_consensus.cc:482] T 0d59c48666ac4e14a2eb9c77d9d443bf P 4fba5977b5054d12b8fee23712c1cd79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:41:20.897377 8430 raft_consensus.cc:504] T 0d59c48666ac4e14a2eb9c77d9d443bf P 4fba5977b5054d12b8fee23712c1cd79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75a048aa032489eaa13cba8d059d8ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42243 } } peers { permanent_uuid: "4fba5977b5054d12b8fee23712c1cd79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41803 } } peers { permanent_uuid: "7a263c43066c4a74b507d658f43b2be3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34955 } }
I20260927 12:41:20.897974 8430 leader_election.cc:288] T 0d59c48666ac4e14a2eb9c77d9d443bf P 4fba5977b5054d12b8fee23712c1cd79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b75a048aa032489eaa13cba8d059d8ab (127.0.98.1:42243), 7a263c43066c4a74b507d658f43b2be3 (127.0.98.3:34955)
I20260927 12:41:20.898409 8068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d59c48666ac4e14a2eb9c77d9d443bf" candidate_uuid: "4fba5977b5054d12b8fee23712c1cd79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b75a048aa032489eaa13cba8d059d8ab" is_pre_election: true
I20260927 12:41:20.898414 8348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d59c48666ac4e14a2eb9c77d9d443bf" candidate_uuid: "4fba5977b5054d12b8fee23712c1cd79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a263c43066c4a74b507d658f43b2be3" is_pre_election: true
I20260927 12:41:20.898566 8068 raft_consensus.cc:2454] T 0d59c48666ac4e14a2eb9c77d9d443bf P b75a048aa032489eaa13cba8d059d8ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fba5977b5054d12b8fee23712c1cd79 in term 0.
I20260927 12:41:20.898571 8348 raft_consensus.cc:2454] T 0d59c48666ac4e14a2eb9c77d9d443bf P 7a263c43066c4a74b507d658f43b2be3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fba5977b5054d12b8fee23712c1cd79 in term 0.
I20260927 12:41:20.898865 8142 leader_election.cc:302] T 0d59c48666ac4e14a2eb9c77d9d443bf P 4fba5977b5054d12b8fee23712c1cd79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fba5977b5054d12b8fee23712c1cd79, 7a263c43066c4a74b507d658f43b2be3; no voters:
I20260927 12:41:20.899085 8430 raft_consensus.cc:2781] T 0d59c48666ac4e14a2eb9c77d9d443bf P 4fba5977b5054d12b8fee23712c1cd79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:41:20.899149 8430 raft_consensus.cc:482] T 0d59c48666ac4e14a2eb9c77d9d443bf P 4fba5977b5054d12b8fee23712c1cd79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:41:20.899202 8430 raft_consensus.cc:3037] T 0d59c48666ac4e14a2eb9c77d9d443bf P 4fba5977b5054d12b8fee23712c1cd79 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:41:20.900170 8430 raft_consensus.cc:504] T 0d59c48666ac4e14a2eb9c77d9d443bf P 4fba5977b5054d12b8fee23712c1cd79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75a048aa032489eaa13cba8d059d8ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42243 } } peers { permanent_uuid: "4fba5977b5054d12b8fee23712c1cd79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41803 } } peers { permanent_uuid: "7a263c43066c4a74b507d658f43b2be3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34955 } }
I20260927 12:41:20.900509 8430 leader_election.cc:288] T 0d59c48666ac4e14a2eb9c77d9d443bf P 4fba5977b5054d12b8fee23712c1cd79 [CANDIDATE]: Term 1 election: Requested vote from peers b75a048aa032489eaa13cba8d059d8ab (127.0.98.1:42243), 7a263c43066c4a74b507d658f43b2be3 (127.0.98.3:34955)
I20260927 12:41:20.900821 8068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d59c48666ac4e14a2eb9c77d9d443bf" candidate_uuid: "4fba5977b5054d12b8fee23712c1cd79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b75a048aa032489eaa13cba8d059d8ab"
I20260927 12:41:20.900843 8348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d59c48666ac4e14a2eb9c77d9d443bf" candidate_uuid: "4fba5977b5054d12b8fee23712c1cd79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a263c43066c4a74b507d658f43b2be3"
I20260927 12:41:20.900960 8348 raft_consensus.cc:3037] T 0d59c48666ac4e14a2eb9c77d9d443bf P 7a263c43066c4a74b507d658f43b2be3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:41:20.900964 8068 raft_consensus.cc:3037] T 0d59c48666ac4e14a2eb9c77d9d443bf P b75a048aa032489eaa13cba8d059d8ab [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:41:20.902014 8068 raft_consensus.cc:2454] T 0d59c48666ac4e14a2eb9c77d9d443bf P b75a048aa032489eaa13cba8d059d8ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fba5977b5054d12b8fee23712c1cd79 in term 1.
I20260927 12:41:20.902127 8348 raft_consensus.cc:2454] T 0d59c48666ac4e14a2eb9c77d9d443bf P 7a263c43066c4a74b507d658f43b2be3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fba5977b5054d12b8fee23712c1cd79 in term 1.
I20260927 12:41:20.902318 8140 leader_election.cc:302] T 0d59c48666ac4e14a2eb9c77d9d443bf P 4fba5977b5054d12b8fee23712c1cd79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fba5977b5054d12b8fee23712c1cd79, b75a048aa032489eaa13cba8d059d8ab; no voters:
I20260927 12:41:20.902633 8430 raft_consensus.cc:2781] T 0d59c48666ac4e14a2eb9c77d9d443bf P 4fba5977b5054d12b8fee23712c1cd79 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:41:20.902724 8430 raft_consensus.cc:686] T 0d59c48666ac4e14a2eb9c77d9d443bf P 4fba5977b5054d12b8fee23712c1cd79 [term 1 LEADER]: Becoming Leader. State: Replica: 4fba5977b5054d12b8fee23712c1cd79, State: Running, Role: LEADER
I20260927 12:41:20.902868 8430 consensus_queue.cc:237] T 0d59c48666ac4e14a2eb9c77d9d443bf P 4fba5977b5054d12b8fee23712c1cd79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75a048aa032489eaa13cba8d059d8ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42243 } } peers { permanent_uuid: "4fba5977b5054d12b8fee23712c1cd79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41803 } } peers { permanent_uuid: "7a263c43066c4a74b507d658f43b2be3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34955 } }
I20260927 12:41:20.904392 7909 catalog_manager.cc:5495] T 0d59c48666ac4e14a2eb9c77d9d443bf P 4fba5977b5054d12b8fee23712c1cd79 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4fba5977b5054d12b8fee23712c1cd79 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "4fba5977b5054d12b8fee23712c1cd79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75a048aa032489eaa13cba8d059d8ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fba5977b5054d12b8fee23712c1cd79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a263c43066c4a74b507d658f43b2be3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34955 } health_report { overall_health: UNKNOWN } } }
Sep 27 12:41:20 dist-test-slave-bnc1 krb5kdc[7866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512879, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 12:41:20.952303 7960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:20.943411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35652 (local address 127.0.98.62:41027)
0927 12:41:20.943553 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:20.943559 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:20.943731 (+ 172us) server_negotiation.cc:407] Connection header received
0927 12:41:20.943940 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:20.943942 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:20.943977 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:20.944060 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:20.944799 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:20.945315 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:20.946219 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:20.946432 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:20.946538 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:20.948982 (+ 2444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:20.948985 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:20.948987 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:41:20.949007 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:20.950480 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:20.951023 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:20.951026 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:20.951027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:20.951091 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:20.951390 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:20.951392 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:20.951393 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:20.951548 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:20.951619 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:20.951962 (+ 343us) server_negotiation.cc:299] Negotiation successful
0927 12:41:20.952148 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":84,"server-negotiator.queue_time_us":53}
I20260927 12:41:20.952327 8457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:20.943336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41027 (local address 127.0.0.1:35652)
0927 12:41:20.943640 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:20.943651 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:20.943805 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:20.944076 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:20.944079 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:20.944573 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:20.944674 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:20.944681 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:20.945437 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:20.945440 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:20.946102 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:20.946111 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:20.946541 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:20.946545 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:20.946620 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:20.947249 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:20.947262 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:20.948834 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:41:20.950592 (+ 1758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:20.950601 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:20.950607 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:20.950927 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:20.951180 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:20.951184 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:20.951186 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:20.951291 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:20.951649 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:41:20.951655 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:41:20.951794 (+ 139us) client_negotiation.cc:769] Sending connection context
0927 12:41:20.951916 (+ 122us) client_negotiation.cc:240] Negotiation successful
0927 12:41:20.952078 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":130,"threads_started":1}
I20260927 12:41:20.956372 7909 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35652:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 12:41:20.957201 7909 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:41:20.962502 8048 tablet_service.cc:1467] Processing CreateTablet for tablet 434bfa0fc2fb4b3e82badc4ea854364a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0e9765e40574476b846b78de408732b1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 12:41:20.962502 8188 tablet_service.cc:1467] Processing CreateTablet for tablet 434bfa0fc2fb4b3e82badc4ea854364a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0e9765e40574476b846b78de408732b1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 12:41:20.962514 8326 tablet_service.cc:1467] Processing CreateTablet for tablet 434bfa0fc2fb4b3e82badc4ea854364a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0e9765e40574476b846b78de408732b1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 12:41:20.962894 8188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 434bfa0fc2fb4b3e82badc4ea854364a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:41:20.962901 8048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 434bfa0fc2fb4b3e82badc4ea854364a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:41:20.962906 8326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 434bfa0fc2fb4b3e82badc4ea854364a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:41:20.965730 8424 tablet_bootstrap.cc:492] T 434bfa0fc2fb4b3e82badc4ea854364a P 4fba5977b5054d12b8fee23712c1cd79: Bootstrap starting.
I20260927 12:41:20.965983 8422 tablet_bootstrap.cc:492] T 434bfa0fc2fb4b3e82badc4ea854364a P b75a048aa032489eaa13cba8d059d8ab: Bootstrap starting.
I20260927 12:41:20.966193 8423 tablet_bootstrap.cc:492] T 434bfa0fc2fb4b3e82badc4ea854364a P 7a263c43066c4a74b507d658f43b2be3: Bootstrap starting.
I20260927 12:41:20.966799 8424 tablet_bootstrap.cc:654] T 434bfa0fc2fb4b3e82badc4ea854364a P 4fba5977b5054d12b8fee23712c1cd79: Neither blocks nor log segments found. Creating new log.
I20260927 12:41:20.966922 8422 tablet_bootstrap.cc:654] T 434bfa0fc2fb4b3e82badc4ea854364a P b75a048aa032489eaa13cba8d059d8ab: Neither blocks nor log segments found. Creating new log.
I20260927 12:41:20.967190 8423 tablet_bootstrap.cc:654] T 434bfa0fc2fb4b3e82badc4ea854364a P 7a263c43066c4a74b507d658f43b2be3: Neither blocks nor log segments found. Creating new log.
I20260927 12:41:20.968612 8424 tablet_bootstrap.cc:492] T 434bfa0fc2fb4b3e82badc4ea854364a P 4fba5977b5054d12b8fee23712c1cd79: No bootstrap required, opened a new log
I20260927 12:41:20.968698 8424 ts_tablet_manager.cc:1397] T 434bfa0fc2fb4b3e82badc4ea854364a P 4fba5977b5054d12b8fee23712c1cd79: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260927 12:41:20.968801 8422 tablet_bootstrap.cc:492] T 434bfa0fc2fb4b3e82badc4ea854364a P b75a048aa032489eaa13cba8d059d8ab: No bootstrap required, opened a new log
I20260927 12:41:20.968832 8423 tablet_bootstrap.cc:492] T 434bfa0fc2fb4b3e82badc4ea854364a P 7a263c43066c4a74b507d658f43b2be3: No bootstrap required, opened a new log
I20260927 12:41:20.968883 8422 ts_tablet_manager.cc:1397] T 434bfa0fc2fb4b3e82badc4ea854364a P b75a048aa032489eaa13cba8d059d8ab: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260927 12:41:20.968914 8423 ts_tablet_manager.cc:1397] T 434bfa0fc2fb4b3e82badc4ea854364a P 7a263c43066c4a74b507d658f43b2be3: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 12:41:20.969144 8424 raft_consensus.cc:348] T 434bfa0fc2fb4b3e82badc4ea854364a P 4fba5977b5054d12b8fee23712c1cd79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75a048aa032489eaa13cba8d059d8ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42243 } } peers { permanent_uuid: "4fba5977b5054d12b8fee23712c1cd79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41803 } } peers { permanent_uuid: "7a263c43066c4a74b507d658f43b2be3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34955 } }
I20260927 12:41:20.969257 8424 raft_consensus.cc:374] T 434bfa0fc2fb4b3e82badc4ea854364a P 4fba5977b5054d12b8fee23712c1cd79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:41:20.969295 8424 raft_consensus.cc:729] T 434bfa0fc2fb4b3e82badc4ea854364a P 4fba5977b5054d12b8fee23712c1cd79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fba5977b5054d12b8fee23712c1cd79, State: Initialized, Role: FOLLOWER
I20260927 12:41:20.969260 8422 raft_consensus.cc:348] T 434bfa0fc2fb4b3e82badc4ea854364a P b75a048aa032489eaa13cba8d059d8ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75a048aa032489eaa13cba8d059d8ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42243 } } peers { permanent_uuid: "4fba5977b5054d12b8fee23712c1cd79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41803 } } peers { permanent_uuid: "7a263c43066c4a74b507d658f43b2be3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34955 } }
I20260927 12:41:20.969357 8422 raft_consensus.cc:374] T 434bfa0fc2fb4b3e82badc4ea854364a P b75a048aa032489eaa13cba8d059d8ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:41:20.969393 8422 raft_consensus.cc:729] T 434bfa0fc2fb4b3e82badc4ea854364a P b75a048aa032489eaa13cba8d059d8ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b75a048aa032489eaa13cba8d059d8ab, State: Initialized, Role: FOLLOWER
I20260927 12:41:20.969358 8423 raft_consensus.cc:348] T 434bfa0fc2fb4b3e82badc4ea854364a P 7a263c43066c4a74b507d658f43b2be3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75a048aa032489eaa13cba8d059d8ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42243 } } peers { permanent_uuid: "4fba5977b5054d12b8fee23712c1cd79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41803 } } peers { permanent_uuid: "7a263c43066c4a74b507d658f43b2be3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34955 } }
I20260927 12:41:20.969470 8423 raft_consensus.cc:374] T 434bfa0fc2fb4b3e82badc4ea854364a P 7a263c43066c4a74b507d658f43b2be3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:41:20.969512 8423 raft_consensus.cc:729] T 434bfa0fc2fb4b3e82badc4ea854364a P 7a263c43066c4a74b507d658f43b2be3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a263c43066c4a74b507d658f43b2be3, State: Initialized, Role: FOLLOWER
I20260927 12:41:20.969516 8422 consensus_queue.cc:260] T 434bfa0fc2fb4b3e82badc4ea854364a P b75a048aa032489eaa13cba8d059d8ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75a048aa032489eaa13cba8d059d8ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42243 } } peers { permanent_uuid: "4fba5977b5054d12b8fee23712c1cd79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41803 } } peers { permanent_uuid: "7a263c43066c4a74b507d658f43b2be3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34955 } }
I20260927 12:41:20.969666 8423 consensus_queue.cc:260] T 434bfa0fc2fb4b3e82badc4ea854364a P 7a263c43066c4a74b507d658f43b2be3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75a048aa032489eaa13cba8d059d8ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42243 } } peers { permanent_uuid: "4fba5977b5054d12b8fee23712c1cd79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41803 } } peers { permanent_uuid: "7a263c43066c4a74b507d658f43b2be3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34955 } }
I20260927 12:41:20.969805 8422 ts_tablet_manager.cc:1428] T 434bfa0fc2fb4b3e82badc4ea854364a P b75a048aa032489eaa13cba8d059d8ab: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 12:41:20.969946 8423 ts_tablet_manager.cc:1428] T 434bfa0fc2fb4b3e82badc4ea854364a P 7a263c43066c4a74b507d658f43b2be3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 12:41:20.969986 8429 tablet_replica.cc:439] T 434bfa0fc2fb4b3e82badc4ea854364a P 7a263c43066c4a74b507d658f43b2be3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75a048aa032489eaa13cba8d059d8ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42243 } } peers { permanent_uuid: "4fba5977b5054d12b8fee23712c1cd79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41803 } } peers { permanent_uuid: "7a263c43066c4a74b507d658f43b2be3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34955 } } }
I20260927 12:41:20.970139 8429 tablet_replica.cc:442] T 434bfa0fc2fb4b3e82badc4ea854364a P 7a263c43066c4a74b507d658f43b2be3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 12:41:20.970096 8428 tablet_replica.cc:439] T 434bfa0fc2fb4b3e82badc4ea854364a P b75a048aa032489eaa13cba8d059d8ab: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75a048aa032489eaa13cba8d059d8ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42243 } } peers { permanent_uuid: "4fba5977b5054d12b8fee23712c1cd79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41803 } } peers { permanent_uuid: "7a263c43066c4a74b507d658f43b2be3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34955 } } }
I20260927 12:41:20.970204 8428 tablet_replica.cc:442] T 434bfa0fc2fb4b3e82badc4ea854364a P b75a048aa032489eaa13cba8d059d8ab: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 12:41:20.970805 8424 consensus_queue.cc:260] T 434bfa0fc2fb4b3e82badc4ea854364a P 4fba5977b5054d12b8fee23712c1cd79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75a048aa032489eaa13cba8d059d8ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42243 } } peers { permanent_uuid: "4fba5977b5054d12b8fee23712c1cd79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41803 } } peers { permanent_uuid: "7a263c43066c4a74b507d658f43b2be3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34955 } }
I20260927 12:41:20.971418 8430 tablet_replica.cc:439] T 434bfa0fc2fb4b3e82badc4ea854364a P 4fba5977b5054d12b8fee23712c1cd79: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75a048aa032489eaa13cba8d059d8ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42243 } } peers { permanent_uuid: "4fba5977b5054d12b8fee23712c1cd79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41803 } } peers { permanent_uuid: "7a263c43066c4a74b507d658f43b2be3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34955 } } }
I20260927 12:41:20.971544 8430 tablet_replica.cc:442] T 434bfa0fc2fb4b3e82badc4ea854364a P 4fba5977b5054d12b8fee23712c1cd79: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 12:41:20.973067 8424 ts_tablet_manager.cc:1428] T 434bfa0fc2fb4b3e82badc4ea854364a P 4fba5977b5054d12b8fee23712c1cd79: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20260927 12:41:21.144815 8428 raft_consensus.cc:482] T 434bfa0fc2fb4b3e82badc4ea854364a P b75a048aa032489eaa13cba8d059d8ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:41:21.145077 8428 raft_consensus.cc:504] T 434bfa0fc2fb4b3e82badc4ea854364a P b75a048aa032489eaa13cba8d059d8ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75a048aa032489eaa13cba8d059d8ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42243 } } peers { permanent_uuid: "4fba5977b5054d12b8fee23712c1cd79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41803 } } peers { permanent_uuid: "7a263c43066c4a74b507d658f43b2be3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34955 } }
I20260927 12:41:21.146289 8428 leader_election.cc:288] T 434bfa0fc2fb4b3e82badc4ea854364a P b75a048aa032489eaa13cba8d059d8ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fba5977b5054d12b8fee23712c1cd79 (127.0.98.2:41803), 7a263c43066c4a74b507d658f43b2be3 (127.0.98.3:34955)
I20260927 12:41:21.150437 8421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:21.146645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41859 (local address 127.0.98.3:34955)
0927 12:41:21.146779 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:21.146785 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:21.147152 (+ 367us) server_negotiation.cc:407] Connection header received
0927 12:41:21.147252 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:21.147256 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:21.147307 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:21.147390 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:41:21.147889 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:21.148403 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:21.149441 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:21.149844 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:21.149923 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:21.150010 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:21.150232 (+ 222us) server_negotiation.cc:299] Negotiation successful
0927 12:41:21.150314 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 12:41:21.150449 8420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:21.146616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52195 (local address 127.0.98.2:41803)
0927 12:41:21.146748 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:21.146753 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:21.147014 (+ 261us) server_negotiation.cc:407] Connection header received
0927 12:41:21.147120 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:21.147123 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:21.147172 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:21.147255 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:41:21.147671 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:21.148277 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:21.149386 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:21.149878 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:21.150009 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:21.150072 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:21.150272 (+ 200us) server_negotiation.cc:299] Negotiation successful
0927 12:41:21.150350 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 12:41:21.150472 8461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:21.146542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34955 (local address 127.0.98.1:41859)
0927 12:41:21.147026 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:21.147041 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:21.147146 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:21.147431 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:21.147433 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:21.147447 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:41:21.147657 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:21.147663 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:21.148516 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:21.148518 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:21.149292 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:21.149300 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:21.149957 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:21.149962 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:21.150065 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:21.150094 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 12:41:21.150206 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 12:41:21.150285 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"mutex_wait_us":3,"thread_start_us":101,"threads_started":1}
I20260927 12:41:21.150861 8462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:21.146537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41803 (local address 127.0.98.1:52195)
0927 12:41:21.146908 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:21.146920 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:21.147023 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:21.147280 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:21.147283 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:21.147298 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:41:21.147435 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:21.147442 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:21.148405 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:21.148408 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:21.149254 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:21.149263 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:21.150034 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:21.150038 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:21.150110 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:21.150127 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 12:41:21.150634 (+ 507us) client_negotiation.cc:240] Negotiation successful
0927 12:41:21.150727 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":118,"threads_started":1}
I20260927 12:41:21.151052 8348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "434bfa0fc2fb4b3e82badc4ea854364a" candidate_uuid: "b75a048aa032489eaa13cba8d059d8ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a263c43066c4a74b507d658f43b2be3" is_pre_election: true
I20260927 12:41:21.151216 8348 raft_consensus.cc:2454] T 434bfa0fc2fb4b3e82badc4ea854364a P 7a263c43066c4a74b507d658f43b2be3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b75a048aa032489eaa13cba8d059d8ab in term 0.
I20260927 12:41:21.151456 8208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "434bfa0fc2fb4b3e82badc4ea854364a" candidate_uuid: "b75a048aa032489eaa13cba8d059d8ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fba5977b5054d12b8fee23712c1cd79" is_pre_election: true
I20260927 12:41:21.151593 8002 leader_election.cc:302] T 434bfa0fc2fb4b3e82badc4ea854364a P b75a048aa032489eaa13cba8d059d8ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a263c43066c4a74b507d658f43b2be3, b75a048aa032489eaa13cba8d059d8ab; no voters:
I20260927 12:41:21.151680 8208 raft_consensus.cc:2454] T 434bfa0fc2fb4b3e82badc4ea854364a P 4fba5977b5054d12b8fee23712c1cd79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b75a048aa032489eaa13cba8d059d8ab in term 0.
I20260927 12:41:21.151861 8428 raft_consensus.cc:2781] T 434bfa0fc2fb4b3e82badc4ea854364a P b75a048aa032489eaa13cba8d059d8ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:41:21.151947 8428 raft_consensus.cc:482] T 434bfa0fc2fb4b3e82badc4ea854364a P b75a048aa032489eaa13cba8d059d8ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:41:21.152001 8428 raft_consensus.cc:3037] T 434bfa0fc2fb4b3e82badc4ea854364a P b75a048aa032489eaa13cba8d059d8ab [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:41:21.152966 8428 raft_consensus.cc:504] T 434bfa0fc2fb4b3e82badc4ea854364a P b75a048aa032489eaa13cba8d059d8ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75a048aa032489eaa13cba8d059d8ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42243 } } peers { permanent_uuid: "4fba5977b5054d12b8fee23712c1cd79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41803 } } peers { permanent_uuid: "7a263c43066c4a74b507d658f43b2be3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34955 } }
I20260927 12:41:21.153342 8428 leader_election.cc:288] T 434bfa0fc2fb4b3e82badc4ea854364a P b75a048aa032489eaa13cba8d059d8ab [CANDIDATE]: Term 1 election: Requested vote from peers 4fba5977b5054d12b8fee23712c1cd79 (127.0.98.2:41803), 7a263c43066c4a74b507d658f43b2be3 (127.0.98.3:34955)
I20260927 12:41:21.153651 8208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "434bfa0fc2fb4b3e82badc4ea854364a" candidate_uuid: "b75a048aa032489eaa13cba8d059d8ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fba5977b5054d12b8fee23712c1cd79"
I20260927 12:41:21.153688 8348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "434bfa0fc2fb4b3e82badc4ea854364a" candidate_uuid: "b75a048aa032489eaa13cba8d059d8ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a263c43066c4a74b507d658f43b2be3"
I20260927 12:41:21.153772 8208 raft_consensus.cc:3037] T 434bfa0fc2fb4b3e82badc4ea854364a P 4fba5977b5054d12b8fee23712c1cd79 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:41:21.153800 8348 raft_consensus.cc:3037] T 434bfa0fc2fb4b3e82badc4ea854364a P 7a263c43066c4a74b507d658f43b2be3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:41:21.154651 8348 raft_consensus.cc:2454] T 434bfa0fc2fb4b3e82badc4ea854364a P 7a263c43066c4a74b507d658f43b2be3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b75a048aa032489eaa13cba8d059d8ab in term 1.
I20260927 12:41:21.154728 8208 raft_consensus.cc:2454] T 434bfa0fc2fb4b3e82badc4ea854364a P 4fba5977b5054d12b8fee23712c1cd79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b75a048aa032489eaa13cba8d059d8ab in term 1.
I20260927 12:41:21.154915 8002 leader_election.cc:302] T 434bfa0fc2fb4b3e82badc4ea854364a P b75a048aa032489eaa13cba8d059d8ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a263c43066c4a74b507d658f43b2be3, b75a048aa032489eaa13cba8d059d8ab; no voters:
I20260927 12:41:21.155088 8428 raft_consensus.cc:2781] T 434bfa0fc2fb4b3e82badc4ea854364a P b75a048aa032489eaa13cba8d059d8ab [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:41:21.155340 8428 raft_consensus.cc:686] T 434bfa0fc2fb4b3e82badc4ea854364a P b75a048aa032489eaa13cba8d059d8ab [term 1 LEADER]: Becoming Leader. State: Replica: b75a048aa032489eaa13cba8d059d8ab, State: Running, Role: LEADER
I20260927 12:41:21.155596 8428 consensus_queue.cc:237] T 434bfa0fc2fb4b3e82badc4ea854364a P b75a048aa032489eaa13cba8d059d8ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75a048aa032489eaa13cba8d059d8ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42243 } } peers { permanent_uuid: "4fba5977b5054d12b8fee23712c1cd79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41803 } } peers { permanent_uuid: "7a263c43066c4a74b507d658f43b2be3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34955 } }
I20260927 12:41:21.156971 8463 tablet_replica.cc:439] T 434bfa0fc2fb4b3e82badc4ea854364a P b75a048aa032489eaa13cba8d059d8ab: TxnStatusTablet state changed. Reason: New leader b75a048aa032489eaa13cba8d059d8ab. Latest consensus state: current_term: 1 leader_uuid: "b75a048aa032489eaa13cba8d059d8ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75a048aa032489eaa13cba8d059d8ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42243 } } peers { permanent_uuid: "4fba5977b5054d12b8fee23712c1cd79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41803 } } peers { permanent_uuid: "7a263c43066c4a74b507d658f43b2be3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34955 } } }
I20260927 12:41:21.157161 8463 tablet_replica.cc:442] T 434bfa0fc2fb4b3e82badc4ea854364a P b75a048aa032489eaa13cba8d059d8ab: This TxnStatusTablet replica's current role is: LEADER
I20260927 12:41:21.157420 8465 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 12:41:21.157569 8465 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 12:41:21.158337 7910 catalog_manager.cc:5495] T 434bfa0fc2fb4b3e82badc4ea854364a P b75a048aa032489eaa13cba8d059d8ab reported cstate change: term changed from 0 to 1, leader changed from <none> to b75a048aa032489eaa13cba8d059d8ab (127.0.98.1). New cstate: current_term: 1 leader_uuid: "b75a048aa032489eaa13cba8d059d8ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75a048aa032489eaa13cba8d059d8ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42243 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fba5977b5054d12b8fee23712c1cd79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a263c43066c4a74b507d658f43b2be3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34955 } health_report { overall_health: UNKNOWN } } }
I20260927 12:41:21.214888 8418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:21.210978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49484 (local address 127.0.98.1:42243)
0927 12:41:21.211121 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:21.211125 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:21.211197 (+ 72us) server_negotiation.cc:407] Connection header received
0927 12:41:21.211429 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:21.211432 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:21.211491 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:21.211613 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:41:21.211853 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:21.212358 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:21.213247 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:21.213541 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:21.213676 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:21.213964 (+ 288us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:41:21.214510 (+ 546us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:41:21.214628 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:21.214694 (+ 66us) server_negotiation.cc:299] Negotiation successful
0927 12:41:21.214745 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 12:41:21.214888 8457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:21.210869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42243 (local address 127.0.0.1:49484)
0927 12:41:21.211138 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:21.211153 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:21.211265 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:21.211621 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:21.211625 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:21.211633 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 12:41:21.211740 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:21.211746 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:21.212487 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:21.212492 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:21.213112 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:21.213120 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:21.213674 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:21.213677 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:21.213764 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:21.213817 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 12:41:21.214599 (+ 782us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 12:41:21.214614 (+ 15us) client_negotiation.cc:769] Sending connection context
0927 12:41:21.214684 (+ 70us) client_negotiation.cc:240] Negotiation successful
0927 12:41:21.214770 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165}
I20260927 12:41:21.217792 8348 raft_consensus.cc:1261] T 434bfa0fc2fb4b3e82badc4ea854364a P 7a263c43066c4a74b507d658f43b2be3 [term 1 FOLLOWER]: Refusing update from remote peer b75a048aa032489eaa13cba8d059d8ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:41:21.217988 8208 raft_consensus.cc:1261] T 434bfa0fc2fb4b3e82badc4ea854364a P 4fba5977b5054d12b8fee23712c1cd79 [term 1 FOLLOWER]: Refusing update from remote peer b75a048aa032489eaa13cba8d059d8ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:41:21.218271 8428 consensus_queue.cc:1035] T 434bfa0fc2fb4b3e82badc4ea854364a P b75a048aa032489eaa13cba8d059d8ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a263c43066c4a74b507d658f43b2be3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:41:21.218518 8463 consensus_queue.cc:1035] T 434bfa0fc2fb4b3e82badc4ea854364a P b75a048aa032489eaa13cba8d059d8ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fba5977b5054d12b8fee23712c1cd79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:41:21.220156 8429 tablet_replica.cc:439] T 434bfa0fc2fb4b3e82badc4ea854364a P 7a263c43066c4a74b507d658f43b2be3: TxnStatusTablet state changed. Reason: New leader b75a048aa032489eaa13cba8d059d8ab. Latest consensus state: current_term: 1 leader_uuid: "b75a048aa032489eaa13cba8d059d8ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75a048aa032489eaa13cba8d059d8ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42243 } } peers { permanent_uuid: "4fba5977b5054d12b8fee23712c1cd79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41803 } } peers { permanent_uuid: "7a263c43066c4a74b507d658f43b2be3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34955 } } }
I20260927 12:41:21.220265 8429 tablet_replica.cc:442] T 434bfa0fc2fb4b3e82badc4ea854364a P 7a263c43066c4a74b507d658f43b2be3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 12:41:21.220420 8430 tablet_replica.cc:439] T 434bfa0fc2fb4b3e82badc4ea854364a P 4fba5977b5054d12b8fee23712c1cd79: TxnStatusTablet state changed. Reason: New leader b75a048aa032489eaa13cba8d059d8ab. Latest consensus state: current_term: 1 leader_uuid: "b75a048aa032489eaa13cba8d059d8ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75a048aa032489eaa13cba8d059d8ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42243 } } peers { permanent_uuid: "4fba5977b5054d12b8fee23712c1cd79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41803 } } peers { permanent_uuid: "7a263c43066c4a74b507d658f43b2be3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34955 } } }
I20260927 12:41:21.220544 8430 tablet_replica.cc:442] T 434bfa0fc2fb4b3e82badc4ea854364a P 4fba5977b5054d12b8fee23712c1cd79: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 12:41:21.221544 8428 tablet_replica.cc:439] T 434bfa0fc2fb4b3e82badc4ea854364a P b75a048aa032489eaa13cba8d059d8ab: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b75a048aa032489eaa13cba8d059d8ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75a048aa032489eaa13cba8d059d8ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42243 } } peers { permanent_uuid: "4fba5977b5054d12b8fee23712c1cd79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41803 } } peers { permanent_uuid: "7a263c43066c4a74b507d658f43b2be3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34955 } } }
I20260927 12:41:21.221797 8428 tablet_replica.cc:442] T 434bfa0fc2fb4b3e82badc4ea854364a P b75a048aa032489eaa13cba8d059d8ab: This TxnStatusTablet replica's current role is: LEADER
I20260927 12:41:21.222641 8429 tablet_replica.cc:439] T 434bfa0fc2fb4b3e82badc4ea854364a P 7a263c43066c4a74b507d658f43b2be3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b75a048aa032489eaa13cba8d059d8ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75a048aa032489eaa13cba8d059d8ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42243 } } peers { permanent_uuid: "4fba5977b5054d12b8fee23712c1cd79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41803 } } peers { permanent_uuid: "7a263c43066c4a74b507d658f43b2be3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34955 } } }
I20260927 12:41:21.222749 8429 tablet_replica.cc:442] T 434bfa0fc2fb4b3e82badc4ea854364a P 7a263c43066c4a74b507d658f43b2be3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 12:41:21.223088 8463 tablet_replica.cc:439] T 434bfa0fc2fb4b3e82badc4ea854364a P b75a048aa032489eaa13cba8d059d8ab: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b75a048aa032489eaa13cba8d059d8ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75a048aa032489eaa13cba8d059d8ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42243 } } peers { permanent_uuid: "4fba5977b5054d12b8fee23712c1cd79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41803 } } peers { permanent_uuid: "7a263c43066c4a74b507d658f43b2be3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34955 } } }
I20260927 12:41:21.223176 8463 tablet_replica.cc:442] T 434bfa0fc2fb4b3e82badc4ea854364a P b75a048aa032489eaa13cba8d059d8ab: This TxnStatusTablet replica's current role is: LEADER
I20260927 12:41:21.225031 8430 tablet_replica.cc:439] T 434bfa0fc2fb4b3e82badc4ea854364a P 4fba5977b5054d12b8fee23712c1cd79: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b75a048aa032489eaa13cba8d059d8ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75a048aa032489eaa13cba8d059d8ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42243 } } peers { permanent_uuid: "4fba5977b5054d12b8fee23712c1cd79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41803 } } peers { permanent_uuid: "7a263c43066c4a74b507d658f43b2be3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34955 } } }
I20260927 12:41:21.225138 8430 tablet_replica.cc:442] T 434bfa0fc2fb4b3e82badc4ea854364a P 4fba5977b5054d12b8fee23712c1cd79: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 12:41:21.237749 8418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:21.234364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53123 (local address 127.0.98.1:42243)
0927 12:41:21.234525 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:21.234532 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:21.234746 (+ 214us) server_negotiation.cc:407] Connection header received
0927 12:41:21.234890 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:21.234893 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:21.234972 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:21.235055 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:41:21.235302 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:21.235834 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:21.236709 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:21.236935 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:21.237039 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:21.237300 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:41:21.237371 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:41:21.237454 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:21.237535 (+ 81us) server_negotiation.cc:299] Negotiation successful
0927 12:41:21.237599 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 12:41:21.237751 8472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:21.234277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42243 (local address 127.0.98.2:53123)
0927 12:41:21.234664 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:21.234675 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:21.234790 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:21.235078 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:21.235081 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:21.235088 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 12:41:21.235191 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:21.235199 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:21.235939 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:21.235952 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:21.236591 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:21.236599 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:21.237039 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:21.237042 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:21.237129 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:21.237175 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 12:41:21.237450 (+ 275us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 12:41:21.237455 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 12:41:21.237517 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 12:41:21.237630 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":114,"threads_started":1}
I20260927 12:41:21.244742 8348 raft_consensus.cc:1261] T 0d59c48666ac4e14a2eb9c77d9d443bf P 7a263c43066c4a74b507d658f43b2be3 [term 1 FOLLOWER]: Refusing update from remote peer 4fba5977b5054d12b8fee23712c1cd79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:41:21.244740 8068 raft_consensus.cc:1261] T 0d59c48666ac4e14a2eb9c77d9d443bf P b75a048aa032489eaa13cba8d059d8ab [term 1 FOLLOWER]: Refusing update from remote peer 4fba5977b5054d12b8fee23712c1cd79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:41:21.245133 8433 consensus_queue.cc:1035] T 0d59c48666ac4e14a2eb9c77d9d443bf P 4fba5977b5054d12b8fee23712c1cd79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b75a048aa032489eaa13cba8d059d8ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:41:21.245255 8430 consensus_queue.cc:1035] T 0d59c48666ac4e14a2eb9c77d9d443bf P 4fba5977b5054d12b8fee23712c1cd79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a263c43066c4a74b507d658f43b2be3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:41:21.250242 8437 mvcc.cc:205] Tried to move back new op lower bound from 7333940761575092224 to 7333940760179404800. Current Snapshot: MvccSnapshot[applied={T|T < 7333940761575092224}]
I20260927 12:41:21.272598 8420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:21.268666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38657 (local address 127.0.98.2:41803)
0927 12:41:21.268792 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:21.268796 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:21.269005 (+ 209us) server_negotiation.cc:407] Connection header received
0927 12:41:21.269165 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:21.269169 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:21.269220 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:21.269310 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:41:21.269624 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:21.270406 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:21.271508 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:21.271743 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:21.271862 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:21.272148 (+ 286us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:41:21.272230 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:41:21.272309 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:21.272396 (+ 87us) server_negotiation.cc:299] Negotiation successful
0927 12:41:21.272451 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 12:41:21.272636 8476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:21.268589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41803 (local address 127.0.98.1:38657)
0927 12:41:21.268915 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:21.268928 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:21.269053 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:21.269319 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:21.269322 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:21.269329 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 12:41:21.269468 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:21.269475 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:21.270516 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:21.270519 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:21.271353 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:21.271382 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:21.271867 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:21.271871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:21.271971 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:21.272014 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 12:41:21.272314 (+ 300us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 12:41:21.272323 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 12:41:21.272386 (+ 63us) client_negotiation.cc:240] Negotiation successful
0927 12:41:21.272474 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":137,"threads_started":1}
I20260927 12:41:21.295799 7910 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35646:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:41:21.296234 7910 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35646:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:41:21.298149 7910 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 83143be63ef94d3b802195c365bfe42a: Sending DeleteTablet for 3 replicas of tablet 0d59c48666ac4e14a2eb9c77d9d443bf
I20260927 12:41:21.298631 8048 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d59c48666ac4e14a2eb9c77d9d443bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:41:21 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49476
I20260927 12:41:21.298735 8326 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d59c48666ac4e14a2eb9c77d9d443bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:41:21 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45500
I20260927 12:41:21.298805 8445 tablet_replica.cc:330] T 0d59c48666ac4e14a2eb9c77d9d443bf P b75a048aa032489eaa13cba8d059d8ab: stopping tablet replica
I20260927 12:41:21.298784 8188 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d59c48666ac4e14a2eb9c77d9d443bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:41:21 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59906
I20260927 12:41:21.298897 8446 tablet_replica.cc:330] T 0d59c48666ac4e14a2eb9c77d9d443bf P 7a263c43066c4a74b507d658f43b2be3: stopping tablet replica
I20260927 12:41:21.298918 8445 raft_consensus.cc:2229] T 0d59c48666ac4e14a2eb9c77d9d443bf P b75a048aa032489eaa13cba8d059d8ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:41:21.298985 8447 tablet_replica.cc:330] T 0d59c48666ac4e14a2eb9c77d9d443bf P 4fba5977b5054d12b8fee23712c1cd79: stopping tablet replica
I20260927 12:41:21.298997 8446 raft_consensus.cc:2229] T 0d59c48666ac4e14a2eb9c77d9d443bf P 7a263c43066c4a74b507d658f43b2be3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:41:21.299062 8445 raft_consensus.cc:2258] T 0d59c48666ac4e14a2eb9c77d9d443bf P b75a048aa032489eaa13cba8d059d8ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:41:21.299093 8446 raft_consensus.cc:2258] T 0d59c48666ac4e14a2eb9c77d9d443bf P 7a263c43066c4a74b507d658f43b2be3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:41:21.299093 8447 raft_consensus.cc:2229] T 0d59c48666ac4e14a2eb9c77d9d443bf P 4fba5977b5054d12b8fee23712c1cd79 [term 1 LEADER]: Raft consensus shutting down.
I20260927 12:41:21.299270 8447 raft_consensus.cc:2258] T 0d59c48666ac4e14a2eb9c77d9d443bf P 4fba5977b5054d12b8fee23712c1cd79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:41:21.299863 8445 ts_tablet_manager.cc:1905] T 0d59c48666ac4e14a2eb9c77d9d443bf P b75a048aa032489eaa13cba8d059d8ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:41:21.299862 8446 ts_tablet_manager.cc:1905] T 0d59c48666ac4e14a2eb9c77d9d443bf P 7a263c43066c4a74b507d658f43b2be3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:41:21.300092 8447 ts_tablet_manager.cc:1905] T 0d59c48666ac4e14a2eb9c77d9d443bf P 4fba5977b5054d12b8fee23712c1cd79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:41:21.302613 8446 ts_tablet_manager.cc:1918] T 0d59c48666ac4e14a2eb9c77d9d443bf P 7a263c43066c4a74b507d658f43b2be3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 12:41:21.302613 8445 ts_tablet_manager.cc:1918] T 0d59c48666ac4e14a2eb9c77d9d443bf P b75a048aa032489eaa13cba8d059d8ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 12:41:21.302680 8445 log.cc:1196] T 0d59c48666ac4e14a2eb9c77d9d443bf P b75a048aa032489eaa13cba8d059d8ab: Deleting WAL directory at /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/minicluster-data/ts-0/wal/wals/0d59c48666ac4e14a2eb9c77d9d443bf
I20260927 12:41:21.302680 8446 log.cc:1196] T 0d59c48666ac4e14a2eb9c77d9d443bf P 7a263c43066c4a74b507d658f43b2be3: Deleting WAL directory at /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/minicluster-data/ts-2/wal/wals/0d59c48666ac4e14a2eb9c77d9d443bf
I20260927 12:41:21.302717 8447 ts_tablet_manager.cc:1918] T 0d59c48666ac4e14a2eb9c77d9d443bf P 4fba5977b5054d12b8fee23712c1cd79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 12:41:21.302780 8447 log.cc:1196] T 0d59c48666ac4e14a2eb9c77d9d443bf P 4fba5977b5054d12b8fee23712c1cd79: Deleting WAL directory at /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/minicluster-data/ts-1/wal/wals/0d59c48666ac4e14a2eb9c77d9d443bf
I20260927 12:41:21.302937 8445 ts_tablet_manager.cc:1939] T 0d59c48666ac4e14a2eb9c77d9d443bf P b75a048aa032489eaa13cba8d059d8ab: Deleting consensus metadata
I20260927 12:41:21.302937 8446 ts_tablet_manager.cc:1939] T 0d59c48666ac4e14a2eb9c77d9d443bf P 7a263c43066c4a74b507d658f43b2be3: Deleting consensus metadata
I20260927 12:41:21.303018 8447 ts_tablet_manager.cc:1939] T 0d59c48666ac4e14a2eb9c77d9d443bf P 4fba5977b5054d12b8fee23712c1cd79: Deleting consensus metadata
I20260927 12:41:21.303571 7898 catalog_manager.cc:4841] TS 7a263c43066c4a74b507d658f43b2be3 (127.0.98.3:34955): tablet 0d59c48666ac4e14a2eb9c77d9d443bf (table test-table [id=7cc68899e87d4c9db9d7a07f2bf7b51e]) successfully deleted
I20260927 12:41:21.303838 7896 catalog_manager.cc:4841] TS b75a048aa032489eaa13cba8d059d8ab (127.0.98.1:42243): tablet 0d59c48666ac4e14a2eb9c77d9d443bf (table test-table [id=7cc68899e87d4c9db9d7a07f2bf7b51e]) successfully deleted
I20260927 12:41:21.304370 7897 catalog_manager.cc:4841] TS 4fba5977b5054d12b8fee23712c1cd79 (127.0.98.2:41803): tablet 0d59c48666ac4e14a2eb9c77d9d443bf (table test-table [id=7cc68899e87d4c9db9d7a07f2bf7b51e]) successfully deleted
Sep 27 12:41:21 dist-test-slave-bnc1 krb5kdc[7866](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 12:41:21 dist-test-slave-bnc1 krb5kdc[7866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:41:21 dist-test-slave-bnc1 krb5kdc[7866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 12:41:21.330094 7960 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:41:21.323200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35656 (local address 127.0.98.62:41027)
0927 12:41:21.323367 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:21.323397 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:21.323619 (+ 222us) server_negotiation.cc:407] Connection header received
0927 12:41:21.323749 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:21.323752 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:21.323798 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:21.323871 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:21.324700 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:21.325209 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:21.326453 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:21.326683 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:21.326807 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:21.329974 (+ 3167us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35656: BlockingRecv error: recv got EOF from 127.0.0.1:35656 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 27 12:41:21 dist-test-slave-bnc1 krb5kdc[7866](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260927 12:41:21.339689 7960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:21.331859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35670 (local address 127.0.98.62:41027)
0927 12:41:21.331970 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:21.331974 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:21.332109 (+ 135us) server_negotiation.cc:407] Connection header received
0927 12:41:21.332249 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:21.332252 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:21.332298 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:21.332380 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:21.332919 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:21.333494 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:21.334348 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:21.334579 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:21.334693 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:21.336754 (+ 2061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:21.336758 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:21.336761 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:41:21.336791 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:21.338078 (+ 1287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:21.338485 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:21.338491 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:21.338492 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:21.338574 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:21.338854 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:21.338857 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:21.338859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:21.339031 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:21.339166 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:21.339405 (+ 239us) server_negotiation.cc:299] Negotiation successful
0927 12:41:21.339528 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 12:41:21.342029 7910 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35670:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 12:41:21.342486 7910 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:41:21.347100 8048 tablet_service.cc:1467] Processing CreateTablet for tablet b854d79af673444599e63b8b37c57ede (DEFAULT_TABLE table=test-table [id=3ba1b8255eaa41ee9ffa473ef81c4ccd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:41:21.347100 8188 tablet_service.cc:1467] Processing CreateTablet for tablet b854d79af673444599e63b8b37c57ede (DEFAULT_TABLE table=test-table [id=3ba1b8255eaa41ee9ffa473ef81c4ccd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:41:21.347285 8326 tablet_service.cc:1467] Processing CreateTablet for tablet b854d79af673444599e63b8b37c57ede (DEFAULT_TABLE table=test-table [id=3ba1b8255eaa41ee9ffa473ef81c4ccd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:41:21.347394 8048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b854d79af673444599e63b8b37c57ede. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:41:21.347415 8188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b854d79af673444599e63b8b37c57ede. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:41:21.347594 8326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b854d79af673444599e63b8b37c57ede. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:41:21.349390 8423 tablet_bootstrap.cc:492] T b854d79af673444599e63b8b37c57ede P 7a263c43066c4a74b507d658f43b2be3: Bootstrap starting.
I20260927 12:41:21.349761 8424 tablet_bootstrap.cc:492] T b854d79af673444599e63b8b37c57ede P 4fba5977b5054d12b8fee23712c1cd79: Bootstrap starting.
I20260927 12:41:21.349771 8422 tablet_bootstrap.cc:492] T b854d79af673444599e63b8b37c57ede P b75a048aa032489eaa13cba8d059d8ab: Bootstrap starting.
I20260927 12:41:21.350363 8423 tablet_bootstrap.cc:654] T b854d79af673444599e63b8b37c57ede P 7a263c43066c4a74b507d658f43b2be3: Neither blocks nor log segments found. Creating new log.
I20260927 12:41:21.350381 8424 tablet_bootstrap.cc:654] T b854d79af673444599e63b8b37c57ede P 4fba5977b5054d12b8fee23712c1cd79: Neither blocks nor log segments found. Creating new log.
I20260927 12:41:21.350574 8422 tablet_bootstrap.cc:654] T b854d79af673444599e63b8b37c57ede P b75a048aa032489eaa13cba8d059d8ab: Neither blocks nor log segments found. Creating new log.
I20260927 12:41:21.351505 8424 tablet_bootstrap.cc:492] T b854d79af673444599e63b8b37c57ede P 4fba5977b5054d12b8fee23712c1cd79: No bootstrap required, opened a new log
I20260927 12:41:21.351631 8424 ts_tablet_manager.cc:1397] T b854d79af673444599e63b8b37c57ede P 4fba5977b5054d12b8fee23712c1cd79: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260927 12:41:21.351713 8423 tablet_bootstrap.cc:492] T b854d79af673444599e63b8b37c57ede P 7a263c43066c4a74b507d658f43b2be3: No bootstrap required, opened a new log
I20260927 12:41:21.351791 8423 ts_tablet_manager.cc:1397] T b854d79af673444599e63b8b37c57ede P 7a263c43066c4a74b507d658f43b2be3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 12:41:21.351994 8424 raft_consensus.cc:348] T b854d79af673444599e63b8b37c57ede P 4fba5977b5054d12b8fee23712c1cd79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75a048aa032489eaa13cba8d059d8ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42243 } } peers { permanent_uuid: "4fba5977b5054d12b8fee23712c1cd79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41803 } } peers { permanent_uuid: "7a263c43066c4a74b507d658f43b2be3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34955 } }
I20260927 12:41:21.352078 8424 raft_consensus.cc:374] T b854d79af673444599e63b8b37c57ede P 4fba5977b5054d12b8fee23712c1cd79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:41:21.352099 8424 raft_consensus.cc:729] T b854d79af673444599e63b8b37c57ede P 4fba5977b5054d12b8fee23712c1cd79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fba5977b5054d12b8fee23712c1cd79, State: Initialized, Role: FOLLOWER
I20260927 12:41:21.352190 8424 consensus_queue.cc:260] T b854d79af673444599e63b8b37c57ede P 4fba5977b5054d12b8fee23712c1cd79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75a048aa032489eaa13cba8d059d8ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42243 } } peers { permanent_uuid: "4fba5977b5054d12b8fee23712c1cd79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41803 } } peers { permanent_uuid: "7a263c43066c4a74b507d658f43b2be3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34955 } }
I20260927 12:41:21.352182 8423 raft_consensus.cc:348] T b854d79af673444599e63b8b37c57ede P 7a263c43066c4a74b507d658f43b2be3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75a048aa032489eaa13cba8d059d8ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42243 } } peers { permanent_uuid: "4fba5977b5054d12b8fee23712c1cd79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41803 } } peers { permanent_uuid: "7a263c43066c4a74b507d658f43b2be3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34955 } }
I20260927 12:41:21.352294 8423 raft_consensus.cc:374] T b854d79af673444599e63b8b37c57ede P 7a263c43066c4a74b507d658f43b2be3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:41:21.352329 8423 raft_consensus.cc:729] T b854d79af673444599e63b8b37c57ede P 7a263c43066c4a74b507d658f43b2be3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a263c43066c4a74b507d658f43b2be3, State: Initialized, Role: FOLLOWER
I20260927 12:41:21.352412 8424 ts_tablet_manager.cc:1428] T b854d79af673444599e63b8b37c57ede P 4fba5977b5054d12b8fee23712c1cd79: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 12:41:21.352464 8423 consensus_queue.cc:260] T b854d79af673444599e63b8b37c57ede P 7a263c43066c4a74b507d658f43b2be3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75a048aa032489eaa13cba8d059d8ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42243 } } peers { permanent_uuid: "4fba5977b5054d12b8fee23712c1cd79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41803 } } peers { permanent_uuid: "7a263c43066c4a74b507d658f43b2be3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34955 } }
I20260927 12:41:21.352722 8423 ts_tablet_manager.cc:1428] T b854d79af673444599e63b8b37c57ede P 7a263c43066c4a74b507d658f43b2be3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 12:41:21.352962 8422 tablet_bootstrap.cc:492] T b854d79af673444599e63b8b37c57ede P b75a048aa032489eaa13cba8d059d8ab: No bootstrap required, opened a new log
I20260927 12:41:21.353070 8422 ts_tablet_manager.cc:1397] T b854d79af673444599e63b8b37c57ede P b75a048aa032489eaa13cba8d059d8ab: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260927 12:41:21.353494 8422 raft_consensus.cc:348] T b854d79af673444599e63b8b37c57ede P b75a048aa032489eaa13cba8d059d8ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75a048aa032489eaa13cba8d059d8ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42243 } } peers { permanent_uuid: "4fba5977b5054d12b8fee23712c1cd79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41803 } } peers { permanent_uuid: "7a263c43066c4a74b507d658f43b2be3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34955 } }
I20260927 12:41:21.353631 8422 raft_consensus.cc:374] T b854d79af673444599e63b8b37c57ede P b75a048aa032489eaa13cba8d059d8ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:41:21.353667 8422 raft_consensus.cc:729] T b854d79af673444599e63b8b37c57ede P b75a048aa032489eaa13cba8d059d8ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b75a048aa032489eaa13cba8d059d8ab, State: Initialized, Role: FOLLOWER
I20260927 12:41:21.353788 8422 consensus_queue.cc:260] T b854d79af673444599e63b8b37c57ede P b75a048aa032489eaa13cba8d059d8ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75a048aa032489eaa13cba8d059d8ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42243 } } peers { permanent_uuid: "4fba5977b5054d12b8fee23712c1cd79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41803 } } peers { permanent_uuid: "7a263c43066c4a74b507d658f43b2be3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34955 } }
I20260927 12:41:21.354082 8422 ts_tablet_manager.cc:1428] T b854d79af673444599e63b8b37c57ede P b75a048aa032489eaa13cba8d059d8ab: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 12:41:21.694361 8477 raft_consensus.cc:482] T b854d79af673444599e63b8b37c57ede P 4fba5977b5054d12b8fee23712c1cd79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:41:21.694532 8477 raft_consensus.cc:504] T b854d79af673444599e63b8b37c57ede P 4fba5977b5054d12b8fee23712c1cd79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75a048aa032489eaa13cba8d059d8ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42243 } } peers { permanent_uuid: "4fba5977b5054d12b8fee23712c1cd79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41803 } } peers { permanent_uuid: "7a263c43066c4a74b507d658f43b2be3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34955 } }
I20260927 12:41:21.695016 8477 leader_election.cc:288] T b854d79af673444599e63b8b37c57ede P 4fba5977b5054d12b8fee23712c1cd79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b75a048aa032489eaa13cba8d059d8ab (127.0.98.1:42243), 7a263c43066c4a74b507d658f43b2be3 (127.0.98.3:34955)
I20260927 12:41:21.695508 8068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b854d79af673444599e63b8b37c57ede" candidate_uuid: "4fba5977b5054d12b8fee23712c1cd79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b75a048aa032489eaa13cba8d059d8ab" is_pre_election: true
I20260927 12:41:21.695528 8348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b854d79af673444599e63b8b37c57ede" candidate_uuid: "4fba5977b5054d12b8fee23712c1cd79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a263c43066c4a74b507d658f43b2be3" is_pre_election: true
I20260927 12:41:21.695695 8068 raft_consensus.cc:2454] T b854d79af673444599e63b8b37c57ede P b75a048aa032489eaa13cba8d059d8ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fba5977b5054d12b8fee23712c1cd79 in term 0.
I20260927 12:41:21.695715 8348 raft_consensus.cc:2454] T b854d79af673444599e63b8b37c57ede P 7a263c43066c4a74b507d658f43b2be3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fba5977b5054d12b8fee23712c1cd79 in term 0.
I20260927 12:41:21.695981 8140 leader_election.cc:302] T b854d79af673444599e63b8b37c57ede P 4fba5977b5054d12b8fee23712c1cd79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fba5977b5054d12b8fee23712c1cd79, b75a048aa032489eaa13cba8d059d8ab; no voters:
I20260927 12:41:21.696198 8477 raft_consensus.cc:2781] T b854d79af673444599e63b8b37c57ede P 4fba5977b5054d12b8fee23712c1cd79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:41:21.696265 8477 raft_consensus.cc:482] T b854d79af673444599e63b8b37c57ede P 4fba5977b5054d12b8fee23712c1cd79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:41:21.696305 8477 raft_consensus.cc:3037] T b854d79af673444599e63b8b37c57ede P 4fba5977b5054d12b8fee23712c1cd79 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:41:21.697438 8477 raft_consensus.cc:504] T b854d79af673444599e63b8b37c57ede P 4fba5977b5054d12b8fee23712c1cd79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75a048aa032489eaa13cba8d059d8ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42243 } } peers { permanent_uuid: "4fba5977b5054d12b8fee23712c1cd79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41803 } } peers { permanent_uuid: "7a263c43066c4a74b507d658f43b2be3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34955 } }
I20260927 12:41:21.697835 8477 leader_election.cc:288] T b854d79af673444599e63b8b37c57ede P 4fba5977b5054d12b8fee23712c1cd79 [CANDIDATE]: Term 1 election: Requested vote from peers b75a048aa032489eaa13cba8d059d8ab (127.0.98.1:42243), 7a263c43066c4a74b507d658f43b2be3 (127.0.98.3:34955)
I20260927 12:41:21.698141 8068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b854d79af673444599e63b8b37c57ede" candidate_uuid: "4fba5977b5054d12b8fee23712c1cd79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b75a048aa032489eaa13cba8d059d8ab"
I20260927 12:41:21.698261 8068 raft_consensus.cc:3037] T b854d79af673444599e63b8b37c57ede P b75a048aa032489eaa13cba8d059d8ab [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:41:21.698230 8348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b854d79af673444599e63b8b37c57ede" candidate_uuid: "4fba5977b5054d12b8fee23712c1cd79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a263c43066c4a74b507d658f43b2be3"
I20260927 12:41:21.698338 8348 raft_consensus.cc:3037] T b854d79af673444599e63b8b37c57ede P 7a263c43066c4a74b507d658f43b2be3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:41:21.699375 8348 raft_consensus.cc:2454] T b854d79af673444599e63b8b37c57ede P 7a263c43066c4a74b507d658f43b2be3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fba5977b5054d12b8fee23712c1cd79 in term 1.
I20260927 12:41:21.699375 8068 raft_consensus.cc:2454] T b854d79af673444599e63b8b37c57ede P b75a048aa032489eaa13cba8d059d8ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fba5977b5054d12b8fee23712c1cd79 in term 1.
I20260927 12:41:21.699651 8142 leader_election.cc:302] T b854d79af673444599e63b8b37c57ede P 4fba5977b5054d12b8fee23712c1cd79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4fba5977b5054d12b8fee23712c1cd79, 7a263c43066c4a74b507d658f43b2be3, b75a048aa032489eaa13cba8d059d8ab; no voters:
I20260927 12:41:21.699792 8477 raft_consensus.cc:2781] T b854d79af673444599e63b8b37c57ede P 4fba5977b5054d12b8fee23712c1cd79 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:41:21.699913 8477 raft_consensus.cc:686] T b854d79af673444599e63b8b37c57ede P 4fba5977b5054d12b8fee23712c1cd79 [term 1 LEADER]: Becoming Leader. State: Replica: 4fba5977b5054d12b8fee23712c1cd79, State: Running, Role: LEADER
I20260927 12:41:21.700078 8477 consensus_queue.cc:237] T b854d79af673444599e63b8b37c57ede P 4fba5977b5054d12b8fee23712c1cd79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75a048aa032489eaa13cba8d059d8ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42243 } } peers { permanent_uuid: "4fba5977b5054d12b8fee23712c1cd79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41803 } } peers { permanent_uuid: "7a263c43066c4a74b507d658f43b2be3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34955 } }
I20260927 12:41:21.701809 7910 catalog_manager.cc:5495] T b854d79af673444599e63b8b37c57ede P 4fba5977b5054d12b8fee23712c1cd79 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4fba5977b5054d12b8fee23712c1cd79 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "4fba5977b5054d12b8fee23712c1cd79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75a048aa032489eaa13cba8d059d8ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fba5977b5054d12b8fee23712c1cd79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a263c43066c4a74b507d658f43b2be3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34955 } health_report { overall_health: UNKNOWN } } }
I20260927 12:41:21.720646 8420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:21.717403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54210 (local address 127.0.98.2:41803)
0927 12:41:21.717526 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:21.717530 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:21.717544 (+ 14us) server_negotiation.cc:407] Connection header received
0927 12:41:21.717713 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:21.717716 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:21.717780 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:21.717879 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:41:21.718131 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:21.718678 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:21.719629 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:21.719837 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:21.719976 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:21.720228 (+ 252us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:41:21.720295 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:41:21.720353 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:21.720429 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 12:41:21.720475 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 12:41:21.722582 8348 raft_consensus.cc:1261] T b854d79af673444599e63b8b37c57ede P 7a263c43066c4a74b507d658f43b2be3 [term 1 FOLLOWER]: Refusing update from remote peer 4fba5977b5054d12b8fee23712c1cd79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:41:21.722580 8068 raft_consensus.cc:1261] T b854d79af673444599e63b8b37c57ede P b75a048aa032489eaa13cba8d059d8ab [term 1 FOLLOWER]: Refusing update from remote peer 4fba5977b5054d12b8fee23712c1cd79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:41:21.723078 8477 consensus_queue.cc:1035] T b854d79af673444599e63b8b37c57ede P 4fba5977b5054d12b8fee23712c1cd79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a263c43066c4a74b507d658f43b2be3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:41:21.723227 8433 consensus_queue.cc:1035] T b854d79af673444599e63b8b37c57ede P 4fba5977b5054d12b8fee23712c1cd79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b75a048aa032489eaa13cba8d059d8ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:41:21.730104 7910 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35670:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:41:21.730232 7910 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35670:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:41:21.732070 7910 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 83143be63ef94d3b802195c365bfe42a: Sending DeleteTablet for 3 replicas of tablet b854d79af673444599e63b8b37c57ede
I20260927 12:41:21.732568 8048 tablet_service.cc:1514] Processing DeleteTablet for tablet b854d79af673444599e63b8b37c57ede with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:41:21 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49476
I20260927 12:41:21.732719 8326 tablet_service.cc:1514] Processing DeleteTablet for tablet b854d79af673444599e63b8b37c57ede with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:41:21 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45500
I20260927 12:41:21.732734 8188 tablet_service.cc:1514] Processing DeleteTablet for tablet b854d79af673444599e63b8b37c57ede with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:41:21 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59906
I20260927 12:41:21.732892 8445 tablet_replica.cc:330] T b854d79af673444599e63b8b37c57ede P b75a048aa032489eaa13cba8d059d8ab: stopping tablet replica
I20260927 12:41:21.732890 8446 tablet_replica.cc:330] T b854d79af673444599e63b8b37c57ede P 7a263c43066c4a74b507d658f43b2be3: stopping tablet replica
I20260927 12:41:21.732988 8446 raft_consensus.cc:2229] T b854d79af673444599e63b8b37c57ede P 7a263c43066c4a74b507d658f43b2be3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:41:21.732988 8445 raft_consensus.cc:2229] T b854d79af673444599e63b8b37c57ede P b75a048aa032489eaa13cba8d059d8ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:41:21.733093 8445 raft_consensus.cc:2258] T b854d79af673444599e63b8b37c57ede P b75a048aa032489eaa13cba8d059d8ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:41:21.733104 8446 raft_consensus.cc:2258] T b854d79af673444599e63b8b37c57ede P 7a263c43066c4a74b507d658f43b2be3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:41:21.733243 8447 tablet_replica.cc:330] T b854d79af673444599e63b8b37c57ede P 4fba5977b5054d12b8fee23712c1cd79: stopping tablet replica
I20260927 12:41:21.733352 8447 raft_consensus.cc:2229] T b854d79af673444599e63b8b37c57ede P 4fba5977b5054d12b8fee23712c1cd79 [term 1 LEADER]: Raft consensus shutting down.
I20260927 12:41:21.733520 8447 raft_consensus.cc:2258] T b854d79af673444599e63b8b37c57ede P 4fba5977b5054d12b8fee23712c1cd79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:41:21.733860 8445 ts_tablet_manager.cc:1905] T b854d79af673444599e63b8b37c57ede P b75a048aa032489eaa13cba8d059d8ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:41:21.734143 8446 ts_tablet_manager.cc:1905] T b854d79af673444599e63b8b37c57ede P 7a263c43066c4a74b507d658f43b2be3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:41:21.734294 8447 ts_tablet_manager.cc:1905] T b854d79af673444599e63b8b37c57ede P 4fba5977b5054d12b8fee23712c1cd79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:41:21.736011 8446 ts_tablet_manager.cc:1918] T b854d79af673444599e63b8b37c57ede P 7a263c43066c4a74b507d658f43b2be3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:41:21.736066 8446 log.cc:1196] T b854d79af673444599e63b8b37c57ede P 7a263c43066c4a74b507d658f43b2be3: Deleting WAL directory at /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/minicluster-data/ts-2/wal/wals/b854d79af673444599e63b8b37c57ede
I20260927 12:41:21.736251 8446 ts_tablet_manager.cc:1939] T b854d79af673444599e63b8b37c57ede P 7a263c43066c4a74b507d658f43b2be3: Deleting consensus metadata
I20260927 12:41:21.736276 8445 ts_tablet_manager.cc:1918] T b854d79af673444599e63b8b37c57ede P b75a048aa032489eaa13cba8d059d8ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:41:21.736344 8445 log.cc:1196] T b854d79af673444599e63b8b37c57ede P b75a048aa032489eaa13cba8d059d8ab: Deleting WAL directory at /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/minicluster-data/ts-0/wal/wals/b854d79af673444599e63b8b37c57ede
I20260927 12:41:21.736603 8445 ts_tablet_manager.cc:1939] T b854d79af673444599e63b8b37c57ede P b75a048aa032489eaa13cba8d059d8ab: Deleting consensus metadata
I20260927 12:41:21.736603 8447 ts_tablet_manager.cc:1918] T b854d79af673444599e63b8b37c57ede P 4fba5977b5054d12b8fee23712c1cd79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:41:21.736661 8447 log.cc:1196] T b854d79af673444599e63b8b37c57ede P 4fba5977b5054d12b8fee23712c1cd79: Deleting WAL directory at /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/minicluster-data/ts-1/wal/wals/b854d79af673444599e63b8b37c57ede
I20260927 12:41:21.736737 7898 catalog_manager.cc:4841] TS 7a263c43066c4a74b507d658f43b2be3 (127.0.98.3:34955): tablet b854d79af673444599e63b8b37c57ede (table test-table [id=3ba1b8255eaa41ee9ffa473ef81c4ccd]) successfully deleted
I20260927 12:41:21.736907 8447 ts_tablet_manager.cc:1939] T b854d79af673444599e63b8b37c57ede P 4fba5977b5054d12b8fee23712c1cd79: Deleting consensus metadata
I20260927 12:41:21.737231 7896 catalog_manager.cc:4841] TS b75a048aa032489eaa13cba8d059d8ab (127.0.98.1:42243): tablet b854d79af673444599e63b8b37c57ede (table test-table [id=3ba1b8255eaa41ee9ffa473ef81c4ccd]) successfully deleted
I20260927 12:41:21.737514 7897 catalog_manager.cc:4841] TS 4fba5977b5054d12b8fee23712c1cd79 (127.0.98.2:41803): tablet b854d79af673444599e63b8b37c57ede (table test-table [id=3ba1b8255eaa41ee9ffa473ef81c4ccd]) successfully deleted
I20260927 12:41:21.737521 7910 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35670:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 12:41:21.737964 7910 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:41:21.742053 8048 tablet_service.cc:1467] Processing CreateTablet for tablet fd44f29cf21546eb9bcbf51cb8e86740 (DEFAULT_TABLE table=test-table [id=6cb1e248f6314322bb0a323bfb7f72fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:41:21.742332 8048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd44f29cf21546eb9bcbf51cb8e86740. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:41:21.742303 8188 tablet_service.cc:1467] Processing CreateTablet for tablet fd44f29cf21546eb9bcbf51cb8e86740 (DEFAULT_TABLE table=test-table [id=6cb1e248f6314322bb0a323bfb7f72fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:41:21.742319 8326 tablet_service.cc:1467] Processing CreateTablet for tablet fd44f29cf21546eb9bcbf51cb8e86740 (DEFAULT_TABLE table=test-table [id=6cb1e248f6314322bb0a323bfb7f72fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:41:21.742511 8188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd44f29cf21546eb9bcbf51cb8e86740. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:41:21.742588 8326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd44f29cf21546eb9bcbf51cb8e86740. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:41:21.744555 8423 tablet_bootstrap.cc:492] T fd44f29cf21546eb9bcbf51cb8e86740 P 7a263c43066c4a74b507d658f43b2be3: Bootstrap starting.
I20260927 12:41:21.744565 8424 tablet_bootstrap.cc:492] T fd44f29cf21546eb9bcbf51cb8e86740 P 4fba5977b5054d12b8fee23712c1cd79: Bootstrap starting.
I20260927 12:41:21.745298 8424 tablet_bootstrap.cc:654] T fd44f29cf21546eb9bcbf51cb8e86740 P 4fba5977b5054d12b8fee23712c1cd79: Neither blocks nor log segments found. Creating new log.
I20260927 12:41:21.745360 8423 tablet_bootstrap.cc:654] T fd44f29cf21546eb9bcbf51cb8e86740 P 7a263c43066c4a74b507d658f43b2be3: Neither blocks nor log segments found. Creating new log.
I20260927 12:41:21.745875 8422 tablet_bootstrap.cc:492] T fd44f29cf21546eb9bcbf51cb8e86740 P b75a048aa032489eaa13cba8d059d8ab: Bootstrap starting.
I20260927 12:41:21.746733 8422 tablet_bootstrap.cc:654] T fd44f29cf21546eb9bcbf51cb8e86740 P b75a048aa032489eaa13cba8d059d8ab: Neither blocks nor log segments found. Creating new log.
I20260927 12:41:21.746742 8423 tablet_bootstrap.cc:492] T fd44f29cf21546eb9bcbf51cb8e86740 P 7a263c43066c4a74b507d658f43b2be3: No bootstrap required, opened a new log
I20260927 12:41:21.746829 8423 ts_tablet_manager.cc:1397] T fd44f29cf21546eb9bcbf51cb8e86740 P 7a263c43066c4a74b507d658f43b2be3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260927 12:41:21.747244 8423 raft_consensus.cc:348] T fd44f29cf21546eb9bcbf51cb8e86740 P 7a263c43066c4a74b507d658f43b2be3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75a048aa032489eaa13cba8d059d8ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42243 } } peers { permanent_uuid: "4fba5977b5054d12b8fee23712c1cd79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41803 } } peers { permanent_uuid: "7a263c43066c4a74b507d658f43b2be3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34955 } }
I20260927 12:41:21.747349 8423 raft_consensus.cc:374] T fd44f29cf21546eb9bcbf51cb8e86740 P 7a263c43066c4a74b507d658f43b2be3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:41:21.747376 8423 raft_consensus.cc:729] T fd44f29cf21546eb9bcbf51cb8e86740 P 7a263c43066c4a74b507d658f43b2be3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a263c43066c4a74b507d658f43b2be3, State: Initialized, Role: FOLLOWER
I20260927 12:41:21.747478 8423 consensus_queue.cc:260] T fd44f29cf21546eb9bcbf51cb8e86740 P 7a263c43066c4a74b507d658f43b2be3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75a048aa032489eaa13cba8d059d8ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42243 } } peers { permanent_uuid: "4fba5977b5054d12b8fee23712c1cd79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41803 } } peers { permanent_uuid: "7a263c43066c4a74b507d658f43b2be3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34955 } }
I20260927 12:41:21.747727 8423 ts_tablet_manager.cc:1428] T fd44f29cf21546eb9bcbf51cb8e86740 P 7a263c43066c4a74b507d658f43b2be3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 12:41:21.748266 8424 tablet_bootstrap.cc:492] T fd44f29cf21546eb9bcbf51cb8e86740 P 4fba5977b5054d12b8fee23712c1cd79: No bootstrap required, opened a new log
I20260927 12:41:21.748353 8424 ts_tablet_manager.cc:1397] T fd44f29cf21546eb9bcbf51cb8e86740 P 4fba5977b5054d12b8fee23712c1cd79: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260927 12:41:21.748798 8424 raft_consensus.cc:348] T fd44f29cf21546eb9bcbf51cb8e86740 P 4fba5977b5054d12b8fee23712c1cd79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75a048aa032489eaa13cba8d059d8ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42243 } } peers { permanent_uuid: "4fba5977b5054d12b8fee23712c1cd79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41803 } } peers { permanent_uuid: "7a263c43066c4a74b507d658f43b2be3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34955 } }
I20260927 12:41:21.748911 8424 raft_consensus.cc:374] T fd44f29cf21546eb9bcbf51cb8e86740 P 4fba5977b5054d12b8fee23712c1cd79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:41:21.748944 8424 raft_consensus.cc:729] T fd44f29cf21546eb9bcbf51cb8e86740 P 4fba5977b5054d12b8fee23712c1cd79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fba5977b5054d12b8fee23712c1cd79, State: Initialized, Role: FOLLOWER
I20260927 12:41:21.749065 8424 consensus_queue.cc:260] T fd44f29cf21546eb9bcbf51cb8e86740 P 4fba5977b5054d12b8fee23712c1cd79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75a048aa032489eaa13cba8d059d8ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42243 } } peers { permanent_uuid: "4fba5977b5054d12b8fee23712c1cd79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41803 } } peers { permanent_uuid: "7a263c43066c4a74b507d658f43b2be3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34955 } }
I20260927 12:41:21.749310 8424 ts_tablet_manager.cc:1428] T fd44f29cf21546eb9bcbf51cb8e86740 P 4fba5977b5054d12b8fee23712c1cd79: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260927 12:41:21.750029 8422 tablet_bootstrap.cc:492] T fd44f29cf21546eb9bcbf51cb8e86740 P b75a048aa032489eaa13cba8d059d8ab: No bootstrap required, opened a new log
I20260927 12:41:21.750106 8422 ts_tablet_manager.cc:1397] T fd44f29cf21546eb9bcbf51cb8e86740 P b75a048aa032489eaa13cba8d059d8ab: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 12:41:21.750449 8422 raft_consensus.cc:348] T fd44f29cf21546eb9bcbf51cb8e86740 P b75a048aa032489eaa13cba8d059d8ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75a048aa032489eaa13cba8d059d8ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42243 } } peers { permanent_uuid: "4fba5977b5054d12b8fee23712c1cd79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41803 } } peers { permanent_uuid: "7a263c43066c4a74b507d658f43b2be3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34955 } }
I20260927 12:41:21.750541 8422 raft_consensus.cc:374] T fd44f29cf21546eb9bcbf51cb8e86740 P b75a048aa032489eaa13cba8d059d8ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:41:21.750571 8422 raft_consensus.cc:729] T fd44f29cf21546eb9bcbf51cb8e86740 P b75a048aa032489eaa13cba8d059d8ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b75a048aa032489eaa13cba8d059d8ab, State: Initialized, Role: FOLLOWER
I20260927 12:41:21.750676 8422 consensus_queue.cc:260] T fd44f29cf21546eb9bcbf51cb8e86740 P b75a048aa032489eaa13cba8d059d8ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75a048aa032489eaa13cba8d059d8ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42243 } } peers { permanent_uuid: "4fba5977b5054d12b8fee23712c1cd79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41803 } } peers { permanent_uuid: "7a263c43066c4a74b507d658f43b2be3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34955 } }
I20260927 12:41:21.750891 8422 ts_tablet_manager.cc:1428] T fd44f29cf21546eb9bcbf51cb8e86740 P b75a048aa032489eaa13cba8d059d8ab: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 12:41:21.843454 8429 raft_consensus.cc:482] T fd44f29cf21546eb9bcbf51cb8e86740 P 7a263c43066c4a74b507d658f43b2be3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:41:21.843621 8429 raft_consensus.cc:504] T fd44f29cf21546eb9bcbf51cb8e86740 P 7a263c43066c4a74b507d658f43b2be3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75a048aa032489eaa13cba8d059d8ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42243 } } peers { permanent_uuid: "4fba5977b5054d12b8fee23712c1cd79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41803 } } peers { permanent_uuid: "7a263c43066c4a74b507d658f43b2be3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34955 } }
I20260927 12:41:21.844805 8429 leader_election.cc:288] T fd44f29cf21546eb9bcbf51cb8e86740 P 7a263c43066c4a74b507d658f43b2be3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b75a048aa032489eaa13cba8d059d8ab (127.0.98.1:42243), 4fba5977b5054d12b8fee23712c1cd79 (127.0.98.2:41803)
I20260927 12:41:21.849012 8420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:21.845181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42589 (local address 127.0.98.2:41803)
0927 12:41:21.845355 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:21.845362 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:21.845467 (+ 105us) server_negotiation.cc:407] Connection header received
0927 12:41:21.845643 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:21.845645 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:21.845684 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:21.845773 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:41:21.846077 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:21.846634 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:21.848002 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:21.848384 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:21.848484 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:21.848551 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:21.848838 (+ 287us) server_negotiation.cc:299] Negotiation successful
0927 12:41:21.848912 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260927 12:41:21.849026 8501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:21.844997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42243 (local address 127.0.98.3:48585)
0927 12:41:21.845302 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:21.845318 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:21.845422 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:21.845760 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:21.845765 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:21.845781 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:41:21.845940 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:21.845945 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:21.846788 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:21.846791 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:21.847846 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:21.847854 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:21.848532 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:21.848535 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:21.848674 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:21.848692 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 12:41:21.848751 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 12:41:21.848837 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":28,"thread_start_us":111,"threads_started":1}
I20260927 12:41:21.849014 8503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:21.845053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48585 (local address 127.0.98.1:42243)
0927 12:41:21.845364 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:21.845369 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:21.845388 (+ 19us) server_negotiation.cc:407] Connection header received
0927 12:41:21.845591 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:21.845597 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:21.845661 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:21.845729 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:41:21.846049 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:21.846644 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:21.847965 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:21.848397 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:21.848524 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:21.848593 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:21.848793 (+ 200us) server_negotiation.cc:299] Negotiation successful
0927 12:41:21.848875 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":75,"threads_started":1}
I20260927 12:41:21.849028 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:21.845040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41803 (local address 127.0.98.3:42589)
0927 12:41:21.845370 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:21.845384 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:21.845514 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:21.845801 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:21.845804 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:21.845832 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:41:21.845956 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:21.845961 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:21.846764 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:21.846769 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:21.847851 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:21.847860 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:21.848532 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:21.848536 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:21.848691 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:21.848708 (+ 17us) client_negotiation.cc:769] Sending connection context
0927 12:41:21.848803 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 12:41:21.848867 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":141,"thread_start_us":71,"threads_started":1}
I20260927 12:41:21.849725 8208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd44f29cf21546eb9bcbf51cb8e86740" candidate_uuid: "7a263c43066c4a74b507d658f43b2be3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fba5977b5054d12b8fee23712c1cd79" is_pre_election: true
I20260927 12:41:21.849884 8208 raft_consensus.cc:2454] T fd44f29cf21546eb9bcbf51cb8e86740 P 4fba5977b5054d12b8fee23712c1cd79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a263c43066c4a74b507d658f43b2be3 in term 0.
I20260927 12:41:21.849932 8068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd44f29cf21546eb9bcbf51cb8e86740" candidate_uuid: "7a263c43066c4a74b507d658f43b2be3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b75a048aa032489eaa13cba8d059d8ab" is_pre_election: true
I20260927 12:41:21.850045 8068 raft_consensus.cc:2454] T fd44f29cf21546eb9bcbf51cb8e86740 P b75a048aa032489eaa13cba8d059d8ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a263c43066c4a74b507d658f43b2be3 in term 0.
I20260927 12:41:21.850277 8281 leader_election.cc:302] T fd44f29cf21546eb9bcbf51cb8e86740 P 7a263c43066c4a74b507d658f43b2be3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fba5977b5054d12b8fee23712c1cd79, 7a263c43066c4a74b507d658f43b2be3; no voters:
I20260927 12:41:21.850567 8429 raft_consensus.cc:2781] T fd44f29cf21546eb9bcbf51cb8e86740 P 7a263c43066c4a74b507d658f43b2be3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:41:21.850631 8429 raft_consensus.cc:482] T fd44f29cf21546eb9bcbf51cb8e86740 P 7a263c43066c4a74b507d658f43b2be3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:41:21.850661 8429 raft_consensus.cc:3037] T fd44f29cf21546eb9bcbf51cb8e86740 P 7a263c43066c4a74b507d658f43b2be3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:41:21.851599 8429 raft_consensus.cc:504] T fd44f29cf21546eb9bcbf51cb8e86740 P 7a263c43066c4a74b507d658f43b2be3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75a048aa032489eaa13cba8d059d8ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42243 } } peers { permanent_uuid: "4fba5977b5054d12b8fee23712c1cd79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41803 } } peers { permanent_uuid: "7a263c43066c4a74b507d658f43b2be3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34955 } }
I20260927 12:41:21.851964 8429 leader_election.cc:288] T fd44f29cf21546eb9bcbf51cb8e86740 P 7a263c43066c4a74b507d658f43b2be3 [CANDIDATE]: Term 1 election: Requested vote from peers b75a048aa032489eaa13cba8d059d8ab (127.0.98.1:42243), 4fba5977b5054d12b8fee23712c1cd79 (127.0.98.2:41803)
I20260927 12:41:21.852201 8068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd44f29cf21546eb9bcbf51cb8e86740" candidate_uuid: "7a263c43066c4a74b507d658f43b2be3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b75a048aa032489eaa13cba8d059d8ab"
I20260927 12:41:21.852303 8208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd44f29cf21546eb9bcbf51cb8e86740" candidate_uuid: "7a263c43066c4a74b507d658f43b2be3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fba5977b5054d12b8fee23712c1cd79"
I20260927 12:41:21.852345 8068 raft_consensus.cc:3037] T fd44f29cf21546eb9bcbf51cb8e86740 P b75a048aa032489eaa13cba8d059d8ab [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:41:21.852421 8208 raft_consensus.cc:3037] T fd44f29cf21546eb9bcbf51cb8e86740 P 4fba5977b5054d12b8fee23712c1cd79 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:41:21.853207 8068 raft_consensus.cc:2454] T fd44f29cf21546eb9bcbf51cb8e86740 P b75a048aa032489eaa13cba8d059d8ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a263c43066c4a74b507d658f43b2be3 in term 1.
I20260927 12:41:21.853377 8208 raft_consensus.cc:2454] T fd44f29cf21546eb9bcbf51cb8e86740 P 4fba5977b5054d12b8fee23712c1cd79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a263c43066c4a74b507d658f43b2be3 in term 1.
I20260927 12:41:21.853462 8280 leader_election.cc:302] T fd44f29cf21546eb9bcbf51cb8e86740 P 7a263c43066c4a74b507d658f43b2be3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a263c43066c4a74b507d658f43b2be3, b75a048aa032489eaa13cba8d059d8ab; no voters:
I20260927 12:41:21.853643 8429 raft_consensus.cc:2781] T fd44f29cf21546eb9bcbf51cb8e86740 P 7a263c43066c4a74b507d658f43b2be3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:41:21.853855 8429 raft_consensus.cc:686] T fd44f29cf21546eb9bcbf51cb8e86740 P 7a263c43066c4a74b507d658f43b2be3 [term 1 LEADER]: Becoming Leader. State: Replica: 7a263c43066c4a74b507d658f43b2be3, State: Running, Role: LEADER
I20260927 12:41:21.854094 8429 consensus_queue.cc:237] T fd44f29cf21546eb9bcbf51cb8e86740 P 7a263c43066c4a74b507d658f43b2be3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75a048aa032489eaa13cba8d059d8ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42243 } } peers { permanent_uuid: "4fba5977b5054d12b8fee23712c1cd79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41803 } } peers { permanent_uuid: "7a263c43066c4a74b507d658f43b2be3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34955 } }
I20260927 12:41:21.856204 7910 catalog_manager.cc:5495] T fd44f29cf21546eb9bcbf51cb8e86740 P 7a263c43066c4a74b507d658f43b2be3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a263c43066c4a74b507d658f43b2be3 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "7a263c43066c4a74b507d658f43b2be3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b75a048aa032489eaa13cba8d059d8ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fba5977b5054d12b8fee23712c1cd79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a263c43066c4a74b507d658f43b2be3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34955 } health_report { overall_health: HEALTHY } } }
I20260927 12:41:21.879290 8508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:21.875819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43866 (local address 127.0.98.3:34955)
0927 12:41:21.876092 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:21.876099 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:21.876111 (+ 12us) server_negotiation.cc:407] Connection header received
0927 12:41:21.876176 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:21.876178 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:21.876225 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:21.876327 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:41:21.876495 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:21.877014 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:21.877868 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:21.878102 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:21.878194 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:21.878395 (+ 201us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:41:21.878931 (+ 536us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:41:21.879019 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:21.879117 (+ 98us) server_negotiation.cc:299] Negotiation successful
0927 12:41:21.879164 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":109,"threads_started":1}
I20260927 12:41:21.888872 8503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:21.885646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47567 (local address 127.0.98.1:42243)
0927 12:41:21.885775 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:21.885779 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:21.885866 (+ 87us) server_negotiation.cc:407] Connection header received
0927 12:41:21.886039 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:21.886042 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:21.886083 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:21.886169 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:41:21.886409 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:21.886909 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:21.887817 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:21.888033 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:21.888146 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:21.888442 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:41:21.888510 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:41:21.888600 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:21.888688 (+ 88us) server_negotiation.cc:299] Negotiation successful
0927 12:41:21.888737 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 12:41:21.888883 8511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:21.885466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42243 (local address 127.0.98.3:47567)
0927 12:41:21.885801 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:21.885815 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:21.885910 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:21.886179 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:21.886182 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:21.886189 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 12:41:21.886298 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:21.886304 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:21.887043 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:21.887049 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:21.887697 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:21.887706 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:21.888170 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:21.888173 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:21.888269 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:21.888308 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 12:41:21.888605 (+ 297us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 12:41:21.888614 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 12:41:21.888680 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 12:41:21.888750 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":143,"threads_started":1}
I20260927 12:41:21.894723 8068 raft_consensus.cc:1261] T fd44f29cf21546eb9bcbf51cb8e86740 P b75a048aa032489eaa13cba8d059d8ab [term 1 FOLLOWER]: Refusing update from remote peer 7a263c43066c4a74b507d658f43b2be3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:41:21.894817 8208 raft_consensus.cc:1261] T fd44f29cf21546eb9bcbf51cb8e86740 P 4fba5977b5054d12b8fee23712c1cd79 [term 1 FOLLOWER]: Refusing update from remote peer 7a263c43066c4a74b507d658f43b2be3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:41:21.895210 8429 consensus_queue.cc:1035] T fd44f29cf21546eb9bcbf51cb8e86740 P 7a263c43066c4a74b507d658f43b2be3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b75a048aa032489eaa13cba8d059d8ab" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:41:21.895671 8504 consensus_queue.cc:1035] T fd44f29cf21546eb9bcbf51cb8e86740 P 7a263c43066c4a74b507d658f43b2be3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fba5977b5054d12b8fee23712c1cd79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:41:21.911446 8508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:21.908055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43291 (local address 127.0.98.3:34955)
0927 12:41:21.908180 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:21.908184 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:21.908391 (+ 207us) server_negotiation.cc:407] Connection header received
0927 12:41:21.908512 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:21.908515 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:21.908558 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:21.908655 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:41:21.908944 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:21.909553 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:21.910406 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:21.910669 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:21.910767 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:21.911029 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:41:21.911105 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:41:21.911184 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:21.911273 (+ 89us) server_negotiation.cc:299] Negotiation successful
0927 12:41:21.911319 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 12:41:21.911469 8515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:21.908001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34955 (local address 127.0.98.1:43291)
0927 12:41:21.908281 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:21.908292 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:21.908409 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:21.908663 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:21.908666 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:21.908673 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 12:41:21.908773 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:21.908778 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:21.909692 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:21.909695 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:21.910311 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:21.910319 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:21.910783 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:21.910789 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:21.910889 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:21.910914 (+ 25us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 12:41:21.911198 (+ 284us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 12:41:21.911205 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 12:41:21.911264 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 12:41:21.911343 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":120,"threads_started":1}
I20260927 12:41:21.931324 7910 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35670:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:41:21.931521 7910 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35670:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:41:21.933756 7910 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 83143be63ef94d3b802195c365bfe42a: Sending DeleteTablet for 3 replicas of tablet fd44f29cf21546eb9bcbf51cb8e86740
I20260927 12:41:21.934376 8188 tablet_service.cc:1514] Processing DeleteTablet for tablet fd44f29cf21546eb9bcbf51cb8e86740 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:41:21 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59906
I20260927 12:41:21.934577 8447 tablet_replica.cc:330] T fd44f29cf21546eb9bcbf51cb8e86740 P 4fba5977b5054d12b8fee23712c1cd79: stopping tablet replica
I20260927 12:41:21.934603 8048 tablet_service.cc:1514] Processing DeleteTablet for tablet fd44f29cf21546eb9bcbf51cb8e86740 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:41:21 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49476
I20260927 12:41:21.934717 8447 raft_consensus.cc:2229] T fd44f29cf21546eb9bcbf51cb8e86740 P 4fba5977b5054d12b8fee23712c1cd79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:41:21.934801 8445 tablet_replica.cc:330] T fd44f29cf21546eb9bcbf51cb8e86740 P b75a048aa032489eaa13cba8d059d8ab: stopping tablet replica
I20260927 12:41:21.934832 8447 raft_consensus.cc:2258] T fd44f29cf21546eb9bcbf51cb8e86740 P 4fba5977b5054d12b8fee23712c1cd79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:41:21.934930 8445 raft_consensus.cc:2229] T fd44f29cf21546eb9bcbf51cb8e86740 P b75a048aa032489eaa13cba8d059d8ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:41:21.935041 8445 raft_consensus.cc:2258] T fd44f29cf21546eb9bcbf51cb8e86740 P b75a048aa032489eaa13cba8d059d8ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:41:21.935504 8447 ts_tablet_manager.cc:1905] T fd44f29cf21546eb9bcbf51cb8e86740 P 4fba5977b5054d12b8fee23712c1cd79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:41:21.935660 8326 tablet_service.cc:1514] Processing DeleteTablet for tablet fd44f29cf21546eb9bcbf51cb8e86740 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:41:21 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45500
I20260927 12:41:21.935989 8445 ts_tablet_manager.cc:1905] T fd44f29cf21546eb9bcbf51cb8e86740 P b75a048aa032489eaa13cba8d059d8ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:41:21.937628 8447 ts_tablet_manager.cc:1918] T fd44f29cf21546eb9bcbf51cb8e86740 P 4fba5977b5054d12b8fee23712c1cd79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 12:41:21.937712 8447 log.cc:1196] T fd44f29cf21546eb9bcbf51cb8e86740 P 4fba5977b5054d12b8fee23712c1cd79: Deleting WAL directory at /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/minicluster-data/ts-1/wal/wals/fd44f29cf21546eb9bcbf51cb8e86740
I20260927 12:41:21.937700 8446 tablet_replica.cc:330] T fd44f29cf21546eb9bcbf51cb8e86740 P 7a263c43066c4a74b507d658f43b2be3: stopping tablet replica
I20260927 12:41:21.937847 8446 raft_consensus.cc:2229] T fd44f29cf21546eb9bcbf51cb8e86740 P 7a263c43066c4a74b507d658f43b2be3 [term 1 LEADER]: Raft consensus shutting down.
I20260927 12:41:21.937966 8447 ts_tablet_manager.cc:1939] T fd44f29cf21546eb9bcbf51cb8e86740 P 4fba5977b5054d12b8fee23712c1cd79: Deleting consensus metadata
I20260927 12:41:21.938184 8446 raft_consensus.cc:2258] T fd44f29cf21546eb9bcbf51cb8e86740 P 7a263c43066c4a74b507d658f43b2be3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:41:21.938469 8445 ts_tablet_manager.cc:1918] T fd44f29cf21546eb9bcbf51cb8e86740 P b75a048aa032489eaa13cba8d059d8ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 12:41:21.938544 8445 log.cc:1196] T fd44f29cf21546eb9bcbf51cb8e86740 P b75a048aa032489eaa13cba8d059d8ab: Deleting WAL directory at /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/minicluster-data/ts-0/wal/wals/fd44f29cf21546eb9bcbf51cb8e86740
I20260927 12:41:21.938601 7897 catalog_manager.cc:4841] TS 4fba5977b5054d12b8fee23712c1cd79 (127.0.98.2:41803): tablet fd44f29cf21546eb9bcbf51cb8e86740 (table test-table [id=6cb1e248f6314322bb0a323bfb7f72fb]) successfully deleted
I20260927 12:41:21.938788 8445 ts_tablet_manager.cc:1939] T fd44f29cf21546eb9bcbf51cb8e86740 P b75a048aa032489eaa13cba8d059d8ab: Deleting consensus metadata
I20260927 12:41:21.938930 8446 ts_tablet_manager.cc:1905] T fd44f29cf21546eb9bcbf51cb8e86740 P 7a263c43066c4a74b507d658f43b2be3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:41:21.939152 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 7983
I20260927 12:41:21.941190 8446 ts_tablet_manager.cc:1918] T fd44f29cf21546eb9bcbf51cb8e86740 P 7a263c43066c4a74b507d658f43b2be3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 12:41:21.941277 8446 log.cc:1196] T fd44f29cf21546eb9bcbf51cb8e86740 P 7a263c43066c4a74b507d658f43b2be3: Deleting WAL directory at /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790512840836255-392-0/minicluster-data/ts-2/wal/wals/fd44f29cf21546eb9bcbf51cb8e86740
I20260927 12:41:21.941509 8446 ts_tablet_manager.cc:1939] T fd44f29cf21546eb9bcbf51cb8e86740 P 7a263c43066c4a74b507d658f43b2be3: Deleting consensus metadata
I20260927 12:41:21.942250 7898 catalog_manager.cc:4841] TS 7a263c43066c4a74b507d658f43b2be3 (127.0.98.3:34955): tablet fd44f29cf21546eb9bcbf51cb8e86740 (table test-table [id=6cb1e248f6314322bb0a323bfb7f72fb]) successfully deleted
W20260927 12:41:21.949143 7896 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42243 (error 108)
I20260927 12:41:21.950198 8520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:21.949834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42243 (local address 127.0.0.1:55854)
0927 12:41:21.950019 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:21.950073 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":57,"threads_started":1}
I20260927 12:41:21.950270 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 8123
W20260927 12:41:21.951380 7896 catalog_manager.cc:4584] TS b75a048aa032489eaa13cba8d059d8ab (127.0.98.1:42243): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fd44f29cf21546eb9bcbf51cb8e86740: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42243: connect: Connection refused (error 111)
I20260927 12:41:21.958189 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 8263
I20260927 12:41:21.964372 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 7879
2026-09-27T12:41:21Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3557 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790512840836255-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:41:22 dist-test-slave-bnc1 krb5kdc[8523](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:41:22 dist-test-slave-bnc1 krb5kdc[8523](info): set up 2 sockets
Sep 27 12:41:22 dist-test-slave-bnc1 krb5kdc[8523](info): commencing operation
krb5kdc: starting...
W20260927 12:41:23.006263 392 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790512840836255-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790512840836255-392-0/krb5kdc/test-user.keytab.
Sep 27 12:41:23 dist-test-slave-bnc1 krb5kdc[8523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:41:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:41:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:41:23.122103 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790512840836255-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790512840836255-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790512840836255-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42573
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42573
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:41:23.228513 8536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:41:23.228770 8536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:41:23.228809 8536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:41:23.232728 8536 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:41:23.232789 8536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:41:23.232808 8536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:41:23.232824 8536 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:41:23.232839 8536 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:41:23.237942 8536 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790512840836255-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790512840836255-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42573
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8536
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790512840836255-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:41:23.238867 8536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:41:23.239778 8536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:41:23.248071 8541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:23.248107 8536 server_base.cc:1034] running on GCE node
W20260927 12:41:23.248059 8544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:23.248059 8542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:23.248880 8536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:41:23.249966 8536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:41:23.251154 8536 hybrid_clock.cc:648] HybridClock initialized: now 1790512883251125 us; error 42 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-27T12:41:23Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1293 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:41:23 dist-test-slave-bnc1 krb5kdc[8549](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:41:23 dist-test-slave-bnc1 krb5kdc[8549](info): set up 2 sockets
Sep 27 12:41:23 dist-test-slave-bnc1 krb5kdc[8549](info): commencing operation
krb5kdc: starting...
W20260927 12:41:24.294340 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/krb5kdc/test-user.keytab.
Sep 27 12:41:24 dist-test-slave-bnc1 krb5kdc[8549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:41:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:41:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:41:24.416397 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42343
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42343
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:41:24.527426 8562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:41:24.527671 8562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:41:24.527710 8562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:41:24.531618 8562 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:41:24.531679 8562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:41:24.531699 8562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:41:24.531716 8562 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:41:24.531736 8562 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:41:24.536808 8562 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42343
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:41:24.537771 8562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:41:24.538682 8562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 12:41:24.546558 8562 server_base.cc:1034] running on GCE node
W20260927 12:41:24.546514 8567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:24.546486 8570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:24.546496 8568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:24.547364 8562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:41:24.548386 8562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:41:24.549618 8562 hybrid_clock.cc:648] HybridClock initialized: now 1790512884549610 us; error 29 us; skew 500 ppm
Sep 27 12:41:24 dist-test-slave-bnc1 krb5kdc[8549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512884, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:41:24.552225 8562 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:41:24.553051 8562 webserver.cc:460] Webserver started at http://127.0.98.62:46701/ using document root <none> and password file <none>
I20260927 12:41:24.553540 8562 fs_manager.cc:359] Metadata directory not provided
I20260927 12:41:24.553668 8562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:41:24.553834 8562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:41:24.555704 8562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/master-0/data/instance:
uuid: "c1626507fba64abda05327572c353cb1"
format_stamp: "Formatted at 2026-09-27 12:41:24 on dist-test-slave-bnc1"
server_key: "15bfb438f889bb0421eb73f308510deb"
server_key_iv: "4191cd9ab2f424a10d010db5c54b5ebd"
server_key_version: "encryptionkey@0"
I20260927 12:41:24.556188 8562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/master-0/wal/instance:
uuid: "c1626507fba64abda05327572c353cb1"
format_stamp: "Formatted at 2026-09-27 12:41:24 on dist-test-slave-bnc1"
server_key: "15bfb438f889bb0421eb73f308510deb"
server_key_iv: "4191cd9ab2f424a10d010db5c54b5ebd"
server_key_version: "encryptionkey@0"
I20260927 12:41:24.559725 8562 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 12:41:24.561961 8577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:24.563093 8562 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 12:41:24.563185 8562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/master-0/wal
uuid: "c1626507fba64abda05327572c353cb1"
format_stamp: "Formatted at 2026-09-27 12:41:24 on dist-test-slave-bnc1"
server_key: "15bfb438f889bb0421eb73f308510deb"
server_key_iv: "4191cd9ab2f424a10d010db5c54b5ebd"
server_key_version: "encryptionkey@0"
I20260927 12:41:24.563252 8562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:41:24.585170 8562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:41:24.587867 8562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:41:24.588016 8562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:41:24.594827 8562 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42343
I20260927 12:41:24.594861 8629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42343 every 8 connection(s)
I20260927 12:41:24.595718 8562 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:41:24.598383 8630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:41:24.602571 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 8562
I20260927 12:41:24.602712 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/master-0/wal/instance
I20260927 12:41:24.602991 392 external_mini_cluster.cc:1428] Setting key 3f959e12d2a3912e0bc159d9227b27c1
I20260927 12:41:24.604305 8630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1626507fba64abda05327572c353cb1: Bootstrap starting.
I20260927 12:41:24.607177 8630 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1626507fba64abda05327572c353cb1: Neither blocks nor log segments found. Creating new log.
I20260927 12:41:24.608179 8630 log.cc:824] T 00000000000000000000000000000000 P c1626507fba64abda05327572c353cb1: Log is configured to *not* fsync() on all Append() calls
I20260927 12:41:24.610637 8630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1626507fba64abda05327572c353cb1: No bootstrap required, opened a new log
Sep 27 12:41:24 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:41:24.615206 8630 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1626507fba64abda05327572c353cb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1626507fba64abda05327572c353cb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42343 } }
I20260927 12:41:24.615428 8630 raft_consensus.cc:374] T 00000000000000000000000000000000 P c1626507fba64abda05327572c353cb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:41:24.615500 8630 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1626507fba64abda05327572c353cb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1626507fba64abda05327572c353cb1, State: Initialized, Role: FOLLOWER
I20260927 12:41:24.615952 8630 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1626507fba64abda05327572c353cb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1626507fba64abda05327572c353cb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42343 } }
I20260927 12:41:24.616070 8630 raft_consensus.cc:388] T 00000000000000000000000000000000 P c1626507fba64abda05327572c353cb1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:41:24.616138 8630 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1626507fba64abda05327572c353cb1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:41:24.616235 8630 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1626507fba64abda05327572c353cb1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:41:24.617376 8630 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1626507fba64abda05327572c353cb1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1626507fba64abda05327572c353cb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42343 } }
I20260927 12:41:24.617770 8630 leader_election.cc:302] T 00000000000000000000000000000000 P c1626507fba64abda05327572c353cb1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1626507fba64abda05327572c353cb1; no voters:
I20260927 12:41:24.618076 8630 leader_election.cc:288] T 00000000000000000000000000000000 P c1626507fba64abda05327572c353cb1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:41:24.618225 8635 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1626507fba64abda05327572c353cb1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:41:24.618453 8635 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1626507fba64abda05327572c353cb1 [term 1 LEADER]: Becoming Leader. State: Replica: c1626507fba64abda05327572c353cb1, State: Running, Role: LEADER
I20260927 12:41:24.618773 8635 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1626507fba64abda05327572c353cb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1626507fba64abda05327572c353cb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42343 } }
I20260927 12:41:24.619421 8630 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1626507fba64abda05327572c353cb1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:41:24.620204 8633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:24.604386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33126 (local address 127.0.98.62:42343)
0927 12:41:24.604802 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:24.604816 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:24.604864 (+ 48us) server_negotiation.cc:407] Connection header received
0927 12:41:24.605550 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:24.605574 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:24.606126 (+ 552us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:24.606555 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:24.607444 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:24.609264 (+ 1820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:24.610543 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:24.611221 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:24.611378 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:24.614476 (+ 3098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:24.614481 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:24.614494 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:41:24.614522 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:24.616447 (+ 1925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:24.616911 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:24.616916 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:24.616921 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:24.617106 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:24.617500 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:24.617506 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:24.617511 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:24.618091 (+ 580us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:24.618398 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:24.618785 (+ 387us) server_negotiation.cc:299] Negotiation successful
0927 12:41:24.619087 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":123,"threads_started":1}
I20260927 12:41:24.621031 8636 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1626507fba64abda05327572c353cb1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c1626507fba64abda05327572c353cb1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1626507fba64abda05327572c353cb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42343 } } }
I20260927 12:41:24.621157 8636 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1626507fba64abda05327572c353cb1 [sys.catalog]: This master's current role is: LEADER
I20260927 12:41:24.621454 8637 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1626507fba64abda05327572c353cb1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1626507fba64abda05327572c353cb1. Latest consensus state: current_term: 1 leader_uuid: "c1626507fba64abda05327572c353cb1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1626507fba64abda05327572c353cb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42343 } } }
I20260927 12:41:24.622542 8637 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1626507fba64abda05327572c353cb1 [sys.catalog]: This master's current role is: LEADER
W20260927 12:41:24.624383 8649 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c1626507fba64abda05327572c353cb1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 12:41:24.624476 8649 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 12:41:24.624559 8650 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:41:24.625305 8650 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:41:24.630664 8650 catalog_manager.cc:1347] Generated new cluster ID: 8599d0ce98d94fd087fd941be5bf41f3
I20260927 12:41:24.630723 8650 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:41:24.642635 8650 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:41:24.643457 8650 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:41:24.654712 8650 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c1626507fba64abda05327572c353cb1: Generated new TSK 0
I20260927 12:41:24.655553 8650 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 12:41:24.692632 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42343
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:41:24.801980 8656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:41:24.802224 8656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:41:24.802284 8656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:41:24.806193 8656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:41:24.806243 8656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:41:24.806305 8656 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:41:24.811389 8656 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8656
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:41:24.812322 8656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:41:24.813280 8656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:41:24.823045 8662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:24.823038 8661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:24.823040 8664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:24.823273 8656 server_base.cc:1034] running on GCE node
I20260927 12:41:24.823747 8656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:41:24.824388 8656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:41:24.825613 8656 hybrid_clock.cc:648] HybridClock initialized: now 1790512884825597 us; error 31 us; skew 500 ppm
Sep 27 12:41:24 dist-test-slave-bnc1 krb5kdc[8549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512884, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:41:24.828359 8656 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 12:41:24.829234 8656 webserver.cc:460] Webserver started at http://127.0.98.1:45455/ using document root <none> and password file <none>
I20260927 12:41:24.829845 8656 fs_manager.cc:359] Metadata directory not provided
I20260927 12:41:24.829931 8656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:41:24.830150 8656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:41:24.832146 8656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-0/data/instance:
uuid: "ffe5ea1b35154da3bc02bb714064a03d"
format_stamp: "Formatted at 2026-09-27 12:41:24 on dist-test-slave-bnc1"
server_key: "84457eee6cb2c65887536d8fae082a3a"
server_key_iv: "b5179486d12205beb07ae1bd7f70fb61"
server_key_version: "encryptionkey@0"
I20260927 12:41:24.832695 8656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ffe5ea1b35154da3bc02bb714064a03d"
format_stamp: "Formatted at 2026-09-27 12:41:24 on dist-test-slave-bnc1"
server_key: "84457eee6cb2c65887536d8fae082a3a"
server_key_iv: "b5179486d12205beb07ae1bd7f70fb61"
server_key_version: "encryptionkey@0"
I20260927 12:41:24.836488 8656 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 12:41:24.838853 8671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:24.840015 8656 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 12:41:24.840142 8656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-0/wal
uuid: "ffe5ea1b35154da3bc02bb714064a03d"
format_stamp: "Formatted at 2026-09-27 12:41:24 on dist-test-slave-bnc1"
server_key: "84457eee6cb2c65887536d8fae082a3a"
server_key_iv: "b5179486d12205beb07ae1bd7f70fb61"
server_key_version: "encryptionkey@0"
I20260927 12:41:24.840266 8656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:41:24.852948 8656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:41:24.856160 8656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:41:24.856366 8656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:41:24.856940 8656 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:41:24.857806 8656 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:41:24.857870 8656 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:24.857942 8656 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:41:24.857983 8656 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:24.866672 8656 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41361
I20260927 12:41:24.866730 8784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41361 every 8 connection(s)
I20260927 12:41:24.867637 8656 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:41:24.868347 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 8656
I20260927 12:41:24.868512 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-0/wal/instance
I20260927 12:41:24.868821 392 external_mini_cluster.cc:1428] Setting key ae6f54c44698ec72ad7947a584220010
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Sep 27 12:41:24 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512884, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:41:24.886766 8633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:24.869486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48981 (local address 127.0.98.62:42343)
0927 12:41:24.869631 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:24.869635 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:24.872119 (+ 2484us) server_negotiation.cc:407] Connection header received
0927 12:41:24.873490 (+ 1371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:24.873493 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:24.873549 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:24.873655 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:24.874815 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:24.875348 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:24.876536 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:24.876774 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:24.876887 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:24.882440 (+ 5553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:24.882445 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:24.882450 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:41:24.882486 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:24.884384 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:24.885102 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:24.885107 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:24.885110 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:24.885214 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:24.885586 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:24.885589 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:24.885591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:24.885798 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:24.885910 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:24.886456 (+ 546us) server_negotiation.cc:299] Negotiation successful
0927 12:41:24.886592 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 12:41:24.887547 8787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:24.869978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42343 (local address 127.0.98.1:48981)
0927 12:41:24.871969 (+ 1991us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:24.872011 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:24.873281 (+ 1270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:24.873834 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:24.873841 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:24.874192 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:24.874670 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:24.874683 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:24.875477 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:24.875481 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:24.876367 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:24.876375 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:24.876926 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:24.876930 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:24.877098 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:24.877738 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:24.877754 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:24.882240 (+ 4486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:41:24.884542 (+ 2302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:24.884551 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:24.884572 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:24.884958 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:24.885309 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:24.885314 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:24.885316 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:24.885468 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:24.885915 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:41:24.885924 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:41:24.886182 (+ 258us) client_negotiation.cc:769] Sending connection context
0927 12:41:24.886373 (+ 191us) client_negotiation.cc:240] Negotiation successful
0927 12:41:24.886722 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1812,"thread_start_us":136,"threads_started":1}
I20260927 12:41:24.888888 8785 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42343
I20260927 12:41:24.889138 8785 heartbeater.cc:464] Registering TS with master...
I20260927 12:41:24.889688 8785 heartbeater.cc:511] Master 127.0.98.62:42343 requested a full tablet report, sending...
I20260927 12:41:24.891192 8594 ts_manager.cc:194] Registered new tserver with Master: ffe5ea1b35154da3bc02bb714064a03d (127.0.98.1:41361)
I20260927 12:41:24.892364 8594 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48981
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 12:41:24.898026 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42343
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:41:25.009606 8790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:41:25.009861 8790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:41:25.009904 8790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:41:25.014144 8790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:41:25.014199 8790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:41:25.014256 8790 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:41:25.019439 8790 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:41:25.020391 8790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:41:25.021309 8790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:41:25.030552 8798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:25.030529 8796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:25.030577 8795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:25.030714 8790 server_base.cc:1034] running on GCE node
I20260927 12:41:25.031195 8790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:41:25.031785 8790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:41:25.032943 8790 hybrid_clock.cc:648] HybridClock initialized: now 1790512885032926 us; error 28 us; skew 500 ppm
Sep 27 12:41:25 dist-test-slave-bnc1 krb5kdc[8549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:41:25.035725 8790 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 12:41:25.036516 8790 webserver.cc:460] Webserver started at http://127.0.98.2:46559/ using document root <none> and password file <none>
I20260927 12:41:25.037043 8790 fs_manager.cc:359] Metadata directory not provided
I20260927 12:41:25.037127 8790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:41:25.037338 8790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:41:25.039291 8790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-1/data/instance:
uuid: "fdf08f4e0d9e45bc97d7d1e9e9ff2e5e"
format_stamp: "Formatted at 2026-09-27 12:41:25 on dist-test-slave-bnc1"
server_key: "fd0224d05a028b8179f0613552cb3a2f"
server_key_iv: "6259a69a485baccb04cfb777572a8a42"
server_key_version: "encryptionkey@0"
I20260927 12:41:25.039814 8790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-1/wal/instance:
uuid: "fdf08f4e0d9e45bc97d7d1e9e9ff2e5e"
format_stamp: "Formatted at 2026-09-27 12:41:25 on dist-test-slave-bnc1"
server_key: "fd0224d05a028b8179f0613552cb3a2f"
server_key_iv: "6259a69a485baccb04cfb777572a8a42"
server_key_version: "encryptionkey@0"
I20260927 12:41:25.043380 8790 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 12:41:25.045646 8805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:25.046739 8790 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:41:25.046870 8790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-1/wal
uuid: "fdf08f4e0d9e45bc97d7d1e9e9ff2e5e"
format_stamp: "Formatted at 2026-09-27 12:41:25 on dist-test-slave-bnc1"
server_key: "fd0224d05a028b8179f0613552cb3a2f"
server_key_iv: "6259a69a485baccb04cfb777572a8a42"
server_key_version: "encryptionkey@0"
I20260927 12:41:25.046972 8790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:41:25.061822 8790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:41:25.065023 8790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:41:25.065212 8790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:41:25.065986 8790 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:41:25.066778 8790 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:41:25.066831 8790 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:25.067178 8790 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:41:25.067227 8790 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:25.075541 8790 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46449
I20260927 12:41:25.075568 8918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46449 every 8 connection(s)
I20260927 12:41:25.076325 8790 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:41:25.083603 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 8790
I20260927 12:41:25.083734 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-1/wal/instance
I20260927 12:41:25.084050 392 external_mini_cluster.cc:1428] Setting key d7280efa7028a1ab53da4b1f78e11005
Sep 27 12:41:25 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:41:25.090478 8633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:25.078033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41193 (local address 127.0.98.62:42343)
0927 12:41:25.078153 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:25.078157 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:25.078928 (+ 771us) server_negotiation.cc:407] Connection header received
0927 12:41:25.080264 (+ 1336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:25.080267 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:25.080321 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:25.080440 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:25.081463 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:25.082095 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:25.083191 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:25.083487 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:25.083598 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:25.086379 (+ 2781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:25.086382 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:25.086384 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:41:25.086404 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:25.088234 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:25.088732 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:25.088739 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:25.088743 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:25.088834 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:25.089149 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:25.089154 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:25.089158 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:25.089400 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:25.089512 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:25.090108 (+ 596us) server_negotiation.cc:299] Negotiation successful
0927 12:41:25.090239 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 12:41:25.091070 8921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:25.078388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42343 (local address 127.0.98.2:41193)
0927 12:41:25.078791 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:25.078829 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:25.080066 (+ 1237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:25.080602 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:25.080608 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:25.080913 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:25.081324 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:25.081335 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:25.082212 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:25.082215 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:25.083038 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:25.083047 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:25.083634 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:25.083636 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:25.083889 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:25.084449 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:25.084464 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:25.086220 (+ 1756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:41:25.088344 (+ 2124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:25.088349 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:25.088359 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:25.088632 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:25.088929 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:25.088931 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:25.088932 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:25.089053 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:25.089532 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:41:25.089537 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:41:25.089820 (+ 283us) client_negotiation.cc:769] Sending connection context
0927 12:41:25.090034 (+ 214us) client_negotiation.cc:240] Negotiation successful
0927 12:41:25.090285 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":99,"threads_started":1}
I20260927 12:41:25.092126 8919 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42343
I20260927 12:41:25.092353 8919 heartbeater.cc:464] Registering TS with master...
I20260927 12:41:25.092797 8919 heartbeater.cc:511] Master 127.0.98.62:42343 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 12:41:25.093858 8594 ts_manager.cc:194] Registered new tserver with Master: fdf08f4e0d9e45bc97d7d1e9e9ff2e5e (127.0.98.2:46449)
I20260927 12:41:25.094380 8594 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41193
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 12:41:25.112106 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42343
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:41:25.220222 8924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:41:25.220494 8924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:41:25.220577 8924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:41:25.224587 8924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:41:25.224673 8924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:41:25.224768 8924 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:41:25.230024 8924 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8924
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:41:25.231012 8924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:41:25.231966 8924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:41:25.241034 8929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:25.241047 8924 server_base.cc:1034] running on GCE node
W20260927 12:41:25.240998 8932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:25.240998 8930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:25.241801 8924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:41:25.242443 8924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:41:25.243620 8924 hybrid_clock.cc:648] HybridClock initialized: now 1790512885243591 us; error 44 us; skew 500 ppm
Sep 27 12:41:25 dist-test-slave-bnc1 krb5kdc[8549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:41:25.246508 8924 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 12:41:25.247329 8924 webserver.cc:460] Webserver started at http://127.0.98.3:38513/ using document root <none> and password file <none>
I20260927 12:41:25.247859 8924 fs_manager.cc:359] Metadata directory not provided
I20260927 12:41:25.247946 8924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:41:25.248165 8924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:41:25.250234 8924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-2/data/instance:
uuid: "36a672b66fea43698c47f30d86142d71"
format_stamp: "Formatted at 2026-09-27 12:41:25 on dist-test-slave-bnc1"
server_key: "2ca81f8c26b6963d79b68be062df79ad"
server_key_iv: "c3d14fc6f7199a1a661cc2d6bfc55bd9"
server_key_version: "encryptionkey@0"
I20260927 12:41:25.250849 8924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-2/wal/instance:
uuid: "36a672b66fea43698c47f30d86142d71"
format_stamp: "Formatted at 2026-09-27 12:41:25 on dist-test-slave-bnc1"
server_key: "2ca81f8c26b6963d79b68be062df79ad"
server_key_iv: "c3d14fc6f7199a1a661cc2d6bfc55bd9"
server_key_version: "encryptionkey@0"
I20260927 12:41:25.254401 8924 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 12:41:25.256604 8939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:25.257805 8924 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 12:41:25.257941 8924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-2/wal
uuid: "36a672b66fea43698c47f30d86142d71"
format_stamp: "Formatted at 2026-09-27 12:41:25 on dist-test-slave-bnc1"
server_key: "2ca81f8c26b6963d79b68be062df79ad"
server_key_iv: "c3d14fc6f7199a1a661cc2d6bfc55bd9"
server_key_version: "encryptionkey@0"
I20260927 12:41:25.258044 8924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:41:25.275784 8924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:41:25.279094 8924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:41:25.279299 8924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:41:25.279855 8924 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:41:25.280644 8924 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:41:25.280733 8924 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:25.280804 8924 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:41:25.280846 8924 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:25.289551 8924 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40155
I20260927 12:41:25.289638 9052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40155 every 8 connection(s)
I20260927 12:41:25.290485 8924 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:41:25.297865 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 8924
I20260927 12:41:25.297995 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-2/wal/instance
I20260927 12:41:25.298303 392 external_mini_cluster.cc:1428] Setting key 068235a60c9cbc17539ca1ca48f55387
Sep 27 12:41:25 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:41:25.303850 8633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:25.292231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50371 (local address 127.0.98.62:42343)
0927 12:41:25.292377 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:25.292382 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:25.293228 (+ 846us) server_negotiation.cc:407] Connection header received
0927 12:41:25.294608 (+ 1380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:25.294611 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:25.294656 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:25.294731 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:25.295823 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:25.296336 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:25.297407 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:25.297708 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:25.297822 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:25.300582 (+ 2760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:25.300587 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:25.300589 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:41:25.300623 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:25.301870 (+ 1247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:25.302373 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:25.302378 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:25.302380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:25.302460 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:25.302799 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:25.302802 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:25.302803 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:25.302977 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:25.303089 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:25.303583 (+ 494us) server_negotiation.cc:299] Negotiation successful
0927 12:41:25.303696 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 12:41:25.304642 9055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:25.292657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42343 (local address 127.0.98.3:50371)
0927 12:41:25.293077 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:25.293117 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:25.294421 (+ 1304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:25.294889 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:25.294896 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:25.295230 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:25.295700 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:25.295712 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:25.296444 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:25.296446 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:25.297301 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:25.297310 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:25.297895 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:25.297898 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:25.298166 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:25.298800 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:25.298816 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:25.300409 (+ 1593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:41:25.301982 (+ 1573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:25.301987 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:25.302002 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:25.302272 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:25.302560 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:25.302563 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:25.302565 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:25.302701 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:25.303096 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:41:25.303104 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:41:25.303343 (+ 239us) client_negotiation.cc:769] Sending connection context
0927 12:41:25.303543 (+ 200us) client_negotiation.cc:240] Negotiation successful
0927 12:41:25.303869 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":116,"threads_started":1}
I20260927 12:41:25.305661 9053 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42343
I20260927 12:41:25.305863 9053 heartbeater.cc:464] Registering TS with master...
I20260927 12:41:25.306272 9053 heartbeater.cc:511] Master 127.0.98.62:42343 requested a full tablet report, sending...
I20260927 12:41:25.307165 8594 ts_manager.cc:194] Registered new tserver with Master: 36a672b66fea43698c47f30d86142d71 (127.0.98.3:40155)
I20260927 12:41:25.307662 8594 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50371
I20260927 12:41:25.311789 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260927 12:41:25.384924 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 8656
I20260927 12:41:25.390560 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 8790
I20260927 12:41:25.395625 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 8924
I20260927 12:41:25.400902 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 8562
I20260927 12:41:25.406950 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42343
--webserver_interface=127.0.98.62
--webserver_port=46701
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42343
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:41:25.514766 9062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:41:25.515089 9062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:41:25.515177 9062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:41:25.519277 9062 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:41:25.519341 9062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:41:25.519361 9062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:41:25.519416 9062 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:41:25.519449 9062 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:41:25.524677 9062 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42343
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=46701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9062
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:41:25.525764 9062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:41:25.526770 9062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:41:25.535050 9068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:25.535204 9062 server_base.cc:1034] running on GCE node
W20260927 12:41:25.535068 9070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:25.535066 9067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:25.535807 9062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:41:25.536934 9062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:41:25.538121 9062 hybrid_clock.cc:648] HybridClock initialized: now 1790512885538105 us; error 29 us; skew 500 ppm
Sep 27 12:41:25 dist-test-slave-bnc1 krb5kdc[8549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:41:25.540822 9062 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:41:25.541805 9062 webserver.cc:460] Webserver started at http://127.0.98.62:46701/ using document root <none> and password file <none>
I20260927 12:41:25.542362 9062 fs_manager.cc:359] Metadata directory not provided
I20260927 12:41:25.542465 9062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:41:25.546252 9062 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260927 12:41:25.548310 9077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:25.549499 9062 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:41:25.549654 9062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/master-0/wal
uuid: "c1626507fba64abda05327572c353cb1"
format_stamp: "Formatted at 2026-09-27 12:41:24 on dist-test-slave-bnc1"
server_key: "15bfb438f889bb0421eb73f308510deb"
server_key_iv: "4191cd9ab2f424a10d010db5c54b5ebd"
server_key_version: "encryptionkey@0"
I20260927 12:41:25.550189 9062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:41:25.579385 9062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:41:25.582696 9062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:41:25.582901 9062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:41:25.590075 9062 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42343
I20260927 12:41:25.590093 9129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42343 every 8 connection(s)
I20260927 12:41:25.591132 9062 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:41:25.592386 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 9062
I20260927 12:41:25.592808 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:41361
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:42343
--webserver_port=45455
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 12:41:25.595983 9130 sys_catalog.cc:263] Verifying existing consensus state
I20260927 12:41:25.598579 9130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1626507fba64abda05327572c353cb1: Bootstrap starting.
I20260927 12:41:25.621053 9130 log.cc:824] T 00000000000000000000000000000000 P c1626507fba64abda05327572c353cb1: Log is configured to *not* fsync() on all Append() calls
I20260927 12:41:25.629259 9130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1626507fba64abda05327572c353cb1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 12:41:25.629707 9130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1626507fba64abda05327572c353cb1: Bootstrap complete.
I20260927 12:41:25.635103 9130 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1626507fba64abda05327572c353cb1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1626507fba64abda05327572c353cb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42343 } }
I20260927 12:41:25.635598 9130 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1626507fba64abda05327572c353cb1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1626507fba64abda05327572c353cb1, State: Initialized, Role: FOLLOWER
I20260927 12:41:25.636171 9130 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1626507fba64abda05327572c353cb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1626507fba64abda05327572c353cb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42343 } }
I20260927 12:41:25.636349 9130 raft_consensus.cc:388] T 00000000000000000000000000000000 P c1626507fba64abda05327572c353cb1 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:41:25.636435 9130 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1626507fba64abda05327572c353cb1 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:41:25.636535 9130 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1626507fba64abda05327572c353cb1 [term 1 FOLLOWER]: Advancing to term 2
I20260927 12:41:25.639034 9130 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1626507fba64abda05327572c353cb1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1626507fba64abda05327572c353cb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42343 } }
I20260927 12:41:25.639397 9130 leader_election.cc:302] T 00000000000000000000000000000000 P c1626507fba64abda05327572c353cb1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1626507fba64abda05327572c353cb1; no voters:
I20260927 12:41:25.639737 9130 leader_election.cc:288] T 00000000000000000000000000000000 P c1626507fba64abda05327572c353cb1 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 12:41:25.639930 9134 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1626507fba64abda05327572c353cb1 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 12:41:25.640256 9134 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1626507fba64abda05327572c353cb1 [term 2 LEADER]: Becoming Leader. State: Replica: c1626507fba64abda05327572c353cb1, State: Running, Role: LEADER
I20260927 12:41:25.640609 9134 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1626507fba64abda05327572c353cb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1626507fba64abda05327572c353cb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42343 } }
I20260927 12:41:25.640947 9130 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1626507fba64abda05327572c353cb1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:41:25.641968 9135 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1626507fba64abda05327572c353cb1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c1626507fba64abda05327572c353cb1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1626507fba64abda05327572c353cb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42343 } } }
I20260927 12:41:25.642094 9135 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1626507fba64abda05327572c353cb1 [sys.catalog]: This master's current role is: LEADER
I20260927 12:41:25.642473 9136 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1626507fba64abda05327572c353cb1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1626507fba64abda05327572c353cb1. Latest consensus state: current_term: 2 leader_uuid: "c1626507fba64abda05327572c353cb1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1626507fba64abda05327572c353cb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42343 } } }
I20260927 12:41:25.642639 9136 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1626507fba64abda05327572c353cb1 [sys.catalog]: This master's current role is: LEADER
I20260927 12:41:25.645779 9150 catalog_manager.cc:1259] Loaded cluster ID: 8599d0ce98d94fd087fd941be5bf41f3
I20260927 12:41:25.645836 9150 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c1626507fba64abda05327572c353cb1: loading cluster ID for follower catalog manager: success
I20260927 12:41:25.648274 9150 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c1626507fba64abda05327572c353cb1: acquiring CA information for follower catalog manager: success
I20260927 12:41:25.649714 9150 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c1626507fba64abda05327572c353cb1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 12:41:25.649828 9143 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:41:25.650660 9143 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:41:25.651093 9143 catalog_manager.cc:1259] Loaded cluster ID: 8599d0ce98d94fd087fd941be5bf41f3
I20260927 12:41:25.651150 9143 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:41:25.651754 9143 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:41:25.652208 9143 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c1626507fba64abda05327572c353cb1: Loaded TSK: 0
I20260927 12:41:25.653041 9143 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 12:41:25.719543 9132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:41:25.719835 9132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:41:25.719920 9132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:41:25.723996 9132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:41:25.724093 9132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:41:25.724190 9132 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:41:25.729544 9132 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:41361
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=45455
--tserver_master_addrs=127.0.98.62:42343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9132
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:41:25.730619 9132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:41:25.731627 9132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:41:25.741930 9159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:25.741935 9157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:25.741940 9156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:25.742336 9132 server_base.cc:1034] running on GCE node
I20260927 12:41:25.742764 9132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:41:25.743446 9132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:41:25.744632 9132 hybrid_clock.cc:648] HybridClock initialized: now 1790512885744602 us; error 48 us; skew 500 ppm
Sep 27 12:41:25 dist-test-slave-bnc1 krb5kdc[8549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512885, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:41:25.747272 9132 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260927 12:41:25.748176 9132 webserver.cc:460] Webserver started at http://127.0.98.1:45455/ using document root <none> and password file <none>
I20260927 12:41:25.748735 9132 fs_manager.cc:359] Metadata directory not provided
I20260927 12:41:25.748807 9132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:41:25.752579 9132 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:41:25.754678 9166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:25.755870 9132 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:41:25.755956 9132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-0/wal
uuid: "ffe5ea1b35154da3bc02bb714064a03d"
format_stamp: "Formatted at 2026-09-27 12:41:24 on dist-test-slave-bnc1"
server_key: "84457eee6cb2c65887536d8fae082a3a"
server_key_iv: "b5179486d12205beb07ae1bd7f70fb61"
server_key_version: "encryptionkey@0"
I20260927 12:41:25.756489 9132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:41:25.769529 9132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:41:25.772756 9132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:41:25.772966 9132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:41:25.773658 9132 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:41:25.774467 9132 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:41:25.774513 9132 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:25.774552 9132 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:41:25.774605 9132 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:25.785359 9132 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41361
I20260927 12:41:25.785403 9279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41361 every 8 connection(s)
I20260927 12:41:25.786413 9132 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:41:25.789809 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 9132
I20260927 12:41:25.790460 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:46449
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:42343
--webserver_port=46559
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 12:41:25 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:41:25 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 12:41:25.800629 9282 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:41:25.788744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38221 (local address 127.0.98.62:42343)
0927 12:41:25.789273 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:25.789296 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:25.789608 (+ 312us) server_negotiation.cc:407] Connection header received
0927 12:41:25.791809 (+ 2201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:25.791820 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:25.792107 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:25.792382 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:25.793828 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:25.794754 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:25.795892 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:25.796370 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:25.796542 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:25.799924 (+ 3382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38221: BlockingRecv error: recv got EOF from 127.0.98.1:38221 (error 108)
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":121,"threads_started":1}
I20260927 12:41:25.800648 9283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:25.788880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42343 (local address 127.0.98.1:38221)
0927 12:41:25.789388 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:25.789447 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:25.791398 (+ 1951us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:25.792550 (+ 1152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:25.792561 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:25.793109 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:25.793668 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:25.793683 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:25.794891 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:25.794898 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:25.795765 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:25.795774 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:25.796526 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:25.796531 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:25.796729 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:25.797367 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:25.797383 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:25.799729 (+ 2346us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":123,"threads_started":1}
W20260927 12:41:25.801266 9280 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42343 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:42343: Client connection negotiation failed: client connection to 127.0.98.62:42343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 27 12:41:25 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:41:25 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:41:25.806686 9283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:25.801756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42343 (local address 127.0.98.1:56815)
0927 12:41:25.801847 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:25.801862 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:25.801959 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:25.802200 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:25.802203 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:25.802353 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:25.802435 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:25.802439 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:25.803136 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:25.803140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:25.803687 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:25.803694 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:25.804141 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:25.804144 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:25.804242 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:25.804745 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:25.804755 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:25.806545 (+ 1790us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":27}
W20260927 12:41:25.806876 9282 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:41:25.801861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56815 (local address 127.0.98.62:42343)
0927 12:41:25.801995 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:25.802001 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:25.802016 (+ 15us) server_negotiation.cc:407] Connection header received
0927 12:41:25.802050 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:25.802052 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:25.802098 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:25.802188 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:25.802541 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:25.803033 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:25.803802 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:25.804033 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:25.804146 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:25.806659 (+ 2513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56815: BlockingRecv error: recv got EOF from 127.0.98.1:56815 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 27 12:41:25 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:41:25 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:41:25.812366 9283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:25.807193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42343 (local address 127.0.98.1:48443)
0927 12:41:25.807313 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:25.807332 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:25.807443 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:25.807689 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:25.807695 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:25.807882 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:25.807982 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:25.807989 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:25.808637 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:25.808640 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:25.809525 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:25.809535 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:25.809947 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:25.809951 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:25.810039 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:25.810463 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:25.810479 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:25.812230 (+ 1751us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 12:41:25.812404 9282 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:41:25.807235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48443 (local address 127.0.98.62:42343)
0927 12:41:25.807329 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:25.807332 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:25.807400 (+ 68us) server_negotiation.cc:407] Connection header received
0927 12:41:25.807543 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:25.807545 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:25.807580 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:25.807638 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:25.808088 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:25.808540 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:25.809657 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:25.809845 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:25.809916 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:25.812328 (+ 2412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:48443: BlockingRecv error: recv got EOF from 127.0.98.1:48443 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260927 12:41:25.812772 9280 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 12:41:25.906663 9284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:41:25.906929 9284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:41:25.906970 9284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:41:25.910920 9284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:41:25.910983 9284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:41:25.911041 9284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:41:25.916234 9284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:46449
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=46559
--tserver_master_addrs=127.0.98.62:42343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9284
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:41:25.917297 9284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:41:25.918308 9284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:41:25.928124 9289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:25.928093 9290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:25.928121 9292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:25.928198 9284 server_base.cc:1034] running on GCE node
I20260927 12:41:25.928730 9284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:41:25.929404 9284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:41:25.930596 9284 hybrid_clock.cc:648] HybridClock initialized: now 1790512885930562 us; error 50 us; skew 500 ppm
Sep 27 12:41:25 dist-test-slave-bnc1 krb5kdc[8549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512885, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:41:25.933316 9284 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260927 12:41:25.934262 9284 webserver.cc:460] Webserver started at http://127.0.98.2:46559/ using document root <none> and password file <none>
I20260927 12:41:25.934810 9284 fs_manager.cc:359] Metadata directory not provided
I20260927 12:41:25.934914 9284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:41:25.938727 9284 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.003s
I20260927 12:41:25.940757 9299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:25.942204 9284 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 12:41:25.942332 9284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-1/wal
uuid: "fdf08f4e0d9e45bc97d7d1e9e9ff2e5e"
format_stamp: "Formatted at 2026-09-27 12:41:25 on dist-test-slave-bnc1"
server_key: "fd0224d05a028b8179f0613552cb3a2f"
server_key_iv: "6259a69a485baccb04cfb777572a8a42"
server_key_version: "encryptionkey@0"
I20260927 12:41:25.942937 9284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:41:25.959246 9284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:41:25.962450 9284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:41:25.962662 9284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:41:25.963240 9284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:41:25.964033 9284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:41:25.964108 9284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:25.964190 9284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:41:25.964241 9284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:25.973519 9284 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46449
I20260927 12:41:25.973650 9412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46449 every 8 connection(s)
I20260927 12:41:25.974627 9284 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:41:25.978041 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 9284
I20260927 12:41:25.978497 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:40155
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:42343
--webserver_port=38513
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 12:41:25 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:41:25 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260927 12:41:25.987469 9282 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:41:25.976766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49671 (local address 127.0.98.62:42343)
0927 12:41:25.976919 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:25.976923 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:25.977780 (+ 857us) server_negotiation.cc:407] Connection header received
0927 12:41:25.979531 (+ 1751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:25.979535 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:25.979605 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:25.979721 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:25.981158 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:25.981859 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:25.983064 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:25.983305 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:25.983422 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:25.987184 (+ 3762us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:49671: BlockingRecv error: recv got EOF from 127.0.98.2:49671 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 12:41:25.987972 9415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:25.977065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42343 (local address 127.0.98.2:49671)
0927 12:41:25.977602 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:25.977668 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:25.979271 (+ 1603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:25.980040 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:25.980058 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:25.980457 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:25.980971 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:25.980995 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:25.982001 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:25.982005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:25.982921 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:25.982931 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:25.983469 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:25.983473 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:25.983648 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:25.984317 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:25.984334 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:25.986985 (+ 2651us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":127,"threads_started":1}
W20260927 12:41:25.988582 9413 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42343 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:42343: Client connection negotiation failed: client connection to 127.0.98.62:42343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 27 12:41:25 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:41:25 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:41:25.994817 9415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:25.989084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42343 (local address 127.0.98.2:33725)
0927 12:41:25.989225 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:25.989242 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:25.989325 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:25.989632 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:25.989635 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:25.989789 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:25.989884 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:25.989889 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:25.990905 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:25.990907 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:25.991548 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:25.991557 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:25.992169 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:25.992173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:25.992285 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:25.992836 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:25.992847 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:25.994722 (+ 1875us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":55}
W20260927 12:41:25.995271 9282 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:41:25.989172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33725 (local address 127.0.98.62:42343)
0927 12:41:25.989298 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:25.989302 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:25.989318 (+ 16us) server_negotiation.cc:407] Connection header received
0927 12:41:25.989415 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:25.989419 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:25.989466 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:25.989554 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:25.989991 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:25.990781 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:25.991684 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:25.992023 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:25.992139 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:25.994915 (+ 2776us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:33725: BlockingRecv error: recv got EOF from 127.0.98.2:33725 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 27 12:41:25 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:41:25 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:41:26.000861 9415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:25.995362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42343 (local address 127.0.98.2:52201)
0927 12:41:25.995646 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:25.995657 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:25.995726 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:25.995981 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:25.995984 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:25.996116 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:25.996187 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:25.996191 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:25.997173 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:25.997176 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:25.997781 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:25.997789 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:25.998343 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:25.998346 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:25.998401 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:25.998871 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:25.998885 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:26.000769 (+ 1884us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224}
W20260927 12:41:26.001030 9282 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:41:25.995545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52201 (local address 127.0.98.62:42343)
0927 12:41:25.995656 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:25.995660 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:25.995696 (+ 36us) server_negotiation.cc:407] Connection header received
0927 12:41:25.995807 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:25.995810 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:25.995852 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:25.995935 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:25.996290 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:25.997064 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:25.997890 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:25.998237 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:25.998330 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:26.000923 (+ 2593us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:52201: BlockingRecv error: recv got EOF from 127.0.98.2:52201 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260927 12:41:26.001286 9413 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 12:41:26.093652 9416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:41:26.093961 9416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:41:26.094051 9416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:41:26.098551 9416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:41:26.098658 9416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:41:26.098774 9416 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:41:26.104385 9416 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:40155
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=38513
--tserver_master_addrs=127.0.98.62:42343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9416
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:41:26.105532 9416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:41:26.106663 9416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:41:26.116541 9421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:26.116624 9424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:26.116616 9416 server_base.cc:1034] running on GCE node
W20260927 12:41:26.116638 9422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:26.117249 9416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:41:26.117995 9416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:41:26.119195 9416 hybrid_clock.cc:648] HybridClock initialized: now 1790512886119175 us; error 30 us; skew 500 ppm
Sep 27 12:41:26 dist-test-slave-bnc1 krb5kdc[8549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512886, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:41:26.122148 9416 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260927 12:41:26.122994 9416 webserver.cc:460] Webserver started at http://127.0.98.3:38513/ using document root <none> and password file <none>
I20260927 12:41:26.123556 9416 fs_manager.cc:359] Metadata directory not provided
I20260927 12:41:26.123667 9416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:41:26.127604 9416 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260927 12:41:26.129684 9431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:26.130848 9416 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:41:26.130973 9416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-2/wal
uuid: "36a672b66fea43698c47f30d86142d71"
format_stamp: "Formatted at 2026-09-27 12:41:25 on dist-test-slave-bnc1"
server_key: "2ca81f8c26b6963d79b68be062df79ad"
server_key_iv: "c3d14fc6f7199a1a661cc2d6bfc55bd9"
server_key_version: "encryptionkey@0"
I20260927 12:41:26.131492 9416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:41:26.148326 9416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:41:26.151430 9416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:41:26.151609 9416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:41:26.152133 9416 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:41:26.152931 9416 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:41:26.152981 9416 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:26.153060 9416 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:41:26.153111 9416 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:26.163089 9416 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40155
I20260927 12:41:26.163107 9544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40155 every 8 connection(s)
I20260927 12:41:26.164161 9416 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:41:26.165398 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 9416
Sep 27 12:41:26 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:41:26 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:41:26.176277 9282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:26.166706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33132 (local address 127.0.98.62:42343)
0927 12:41:26.167404 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:26.167408 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:26.167422 (+ 14us) server_negotiation.cc:407] Connection header received
0927 12:41:26.167463 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:26.167465 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:26.167501 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:26.167576 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:26.168394 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:26.169271 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:26.170297 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:26.170599 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:26.170709 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:26.172036 (+ 1327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:26.172042 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:26.172057 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:41:26.172089 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:26.174013 (+ 1924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:26.174467 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:26.174474 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:26.174480 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:26.174617 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:26.174901 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:26.174904 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:26.174906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:26.175264 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:26.175428 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:26.175825 (+ 397us) server_negotiation.cc:299] Negotiation successful
0927 12:41:26.176146 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":95,"threads_started":1}
W20260927 12:41:26.177196 9548 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:41:26.166298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51723 (local address 127.0.98.62:42343)
0927 12:41:26.166977 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:26.166984 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:26.168004 (+ 1020us) server_negotiation.cc:407] Connection header received
0927 12:41:26.169615 (+ 1611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:26.169619 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:26.169687 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:26.169783 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:26.171044 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:26.171546 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:26.173107 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:26.173329 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:26.173406 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:26.177069 (+ 3663us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51723: BlockingRecv error: recv got EOF from 127.0.98.3:51723 (error 108)
Metrics: {"server-negotiator.queue_time_us":605}
I20260927 12:41:26.177807 9549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:26.166581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42343 (local address 127.0.98.3:51723)
0927 12:41:26.167870 (+ 1289us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:26.167909 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:26.169384 (+ 1475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:26.169928 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:26.169935 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:26.170297 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:26.170910 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:26.170925 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:26.171682 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:26.171685 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:26.172984 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:26.172997 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:26.173473 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:26.173477 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:26.173706 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:26.174214 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:26.174228 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:26.176906 (+ 2678us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1123,"thread_start_us":127,"threads_started":1}
W20260927 12:41:26.178234 9545 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42343 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:42343: Client connection negotiation failed: client connection to 127.0.98.62:42343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 27 12:41:26 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:41:26 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:41:26.183521 9549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:26.178687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42343 (local address 127.0.98.3:45565)
0927 12:41:26.179037 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:26.179052 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:26.179141 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:26.179386 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:26.179388 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:26.179509 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:26.179589 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:26.179594 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:26.180232 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:26.180235 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:26.180807 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:26.180815 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:26.181187 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:26.181190 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:26.181253 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:26.181736 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:26.181747 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:26.183432 (+ 1685us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":289}
W20260927 12:41:26.183720 9548 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:41:26.178914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45565 (local address 127.0.98.62:42343)
0927 12:41:26.179024 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:26.179028 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:26.179094 (+ 66us) server_negotiation.cc:407] Connection header received
0927 12:41:26.179236 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:26.179239 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:26.179277 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:26.179351 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:26.179681 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:26.180141 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:26.180900 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:26.181095 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:26.181166 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:26.183585 (+ 2419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:45565: BlockingRecv error: recv got EOF from 127.0.98.3:45565 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 27 12:41:26 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:41:26 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:41:26.188526 9549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:26.184091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42343 (local address 127.0.98.3:40191)
0927 12:41:26.184210 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:26.184226 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:26.184300 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:26.184500 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:26.184502 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:26.184660 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:26.184741 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:26.184745 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:26.185392 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:26.185394 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:26.185971 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:26.185978 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:26.186351 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:26.186353 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:26.186406 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:26.186787 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:26.186797 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:26.188444 (+ 1647us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 12:41:26.188709 9548 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:41:26.184170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40191 (local address 127.0.98.62:42343)
0927 12:41:26.184308 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:26.184311 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:26.184323 (+ 12us) server_negotiation.cc:407] Connection header received
0927 12:41:26.184373 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:26.184375 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:26.184406 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:26.184470 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:26.184832 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:26.185307 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:26.186060 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:26.186261 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:26.186336 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:26.188582 (+ 2246us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:40191: BlockingRecv error: recv got EOF from 127.0.98.3:40191 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260927 12:41:26.188731 9545 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 12:41:26 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:41:26 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:41:26.819497 9550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:26.813390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42343 (local address 127.0.98.1:34343)
0927 12:41:26.813673 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:26.813686 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:26.813771 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:26.814027 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:26.814030 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:26.814200 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:26.814310 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:26.814316 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:26.815082 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:26.815085 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:26.815782 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:26.815794 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:26.816282 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:26.816286 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:26.816371 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:26.816891 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:26.816905 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:26.819341 (+ 2436us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":94,"threads_started":1}
W20260927 12:41:26.819732 9551 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:41:26.813513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34343 (local address 127.0.98.62:42343)
0927 12:41:26.813822 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:26.813825 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:26.813838 (+ 13us) server_negotiation.cc:407] Connection header received
0927 12:41:26.813871 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:26.813873 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:26.813912 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:26.813992 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:26.814409 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:26.814941 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:26.815904 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:26.816126 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:26.816203 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:26.819535 (+ 3332us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:34343: BlockingRecv error: recv got EOF from 127.0.98.1:34343 (error 108)
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":151,"threads_started":1}
Sep 27 12:41:27 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:41:27 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:41:27.007648 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:27.002003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42343 (local address 127.0.98.2:57193)
0927 12:41:27.002357 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:27.002371 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:27.002506 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:27.002773 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:27.002776 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:27.002984 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:27.003133 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:27.003141 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:27.004114 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:27.004117 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:27.004710 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:27.004718 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:27.005104 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:27.005107 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:27.005172 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:27.005737 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:27.005752 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:27.007526 (+ 1774us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":85,"threads_started":1}
W20260927 12:41:27.007771 9551 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:41:27.002128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57193 (local address 127.0.98.62:42343)
0927 12:41:27.002290 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:27.002295 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:27.002447 (+ 152us) server_negotiation.cc:407] Connection header received
0927 12:41:27.002607 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:27.002611 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:27.002661 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:27.002750 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:27.003257 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:27.004001 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:27.004806 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:27.005016 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:27.005091 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:27.007659 (+ 2568us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:57193: BlockingRecv error: recv got EOF from 127.0.98.2:57193 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 27 12:41:27 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:41:27 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:41:27.194463 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:27.189368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42343 (local address 127.0.98.3:58439)
0927 12:41:27.189662 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:27.189675 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:27.189772 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:27.190051 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:27.190055 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:27.190261 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:27.190373 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:27.190379 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:27.191077 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:27.191079 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:27.191648 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:27.191656 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:27.192029 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:27.192031 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:27.192097 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:27.192554 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:27.192565 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:27.194334 (+ 1769us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":97,"threads_started":1}
W20260927 12:41:27.194623 9551 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:41:27.189411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58439 (local address 127.0.98.62:42343)
0927 12:41:27.189543 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:27.189548 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:27.189741 (+ 193us) server_negotiation.cc:407] Connection header received
0927 12:41:27.189863 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:27.189867 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:27.189927 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:27.190034 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:27.190472 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:27.190981 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:27.191738 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:27.191942 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:27.192017 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:27.194472 (+ 2455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58439: BlockingRecv error: recv got EOF from 127.0.98.3:58439 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 27 12:41:27 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:41:27 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:41:27.825719 9554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:27.820393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42343 (local address 127.0.98.1:57689)
0927 12:41:27.820667 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:27.820681 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:27.820781 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:27.821012 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:27.821014 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:27.821202 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:27.821299 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:27.821304 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:27.822048 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:27.822051 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:27.822657 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:27.822667 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:27.823093 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:27.823096 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:27.823165 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:27.823637 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:27.823649 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:27.825496 (+ 1847us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":98,"threads_started":1}
W20260927 12:41:27.825814 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:41:27.820435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57689 (local address 127.0.98.62:42343)
0927 12:41:27.820654 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:27.820658 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:27.820748 (+ 90us) server_negotiation.cc:407] Connection header received
0927 12:41:27.820855 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:27.820857 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:27.820899 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:27.820976 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:27.821394 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:27.821926 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:27.822765 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:27.822979 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:27.823051 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:27.825683 (+ 2632us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:57689: BlockingRecv error: recv got EOF from 127.0.98.1:57689 (error 108)
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":80,"threads_started":1}
Sep 27 12:41:28 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:41:28 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:41:28.013926 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:28.008603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42343 (local address 127.0.98.2:47407)
0927 12:41:28.008877 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:28.008890 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:28.008989 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:28.009218 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:28.009220 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:28.009400 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:28.009519 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:28.009525 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:28.010268 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:28.010270 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:28.010841 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:28.010848 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:28.011224 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:28.011227 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:28.011284 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:28.011759 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:28.011771 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:28.013741 (+ 1970us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":102,"threads_started":1}
W20260927 12:41:28.013962 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:41:28.008670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47407 (local address 127.0.98.62:42343)
0927 12:41:28.008805 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:28.008808 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:28.008956 (+ 148us) server_negotiation.cc:407] Connection header received
0927 12:41:28.009056 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:28.009058 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:28.009100 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:28.009186 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:28.009668 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:28.010177 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:28.010934 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:28.011136 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:28.011207 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:28.013866 (+ 2659us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:47407: BlockingRecv error: recv got EOF from 127.0.98.2:47407 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 27 12:41:28 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:41:28 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:41:28.200639 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:28.195229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42343 (local address 127.0.98.3:50239)
0927 12:41:28.195521 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:28.195537 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:28.195655 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:28.195926 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:28.195929 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:28.196127 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:28.196219 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:28.196225 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:28.196986 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:28.196991 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:28.197548 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:28.197556 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:28.198039 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:28.198043 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:28.198132 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:28.198651 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:28.198663 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:28.200429 (+ 1766us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":117,"threads_started":1}
W20260927 12:41:28.200654 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:41:28.195455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50239 (local address 127.0.98.62:42343)
0927 12:41:28.195584 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:28.195590 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:28.195606 (+ 16us) server_negotiation.cc:407] Connection header received
0927 12:41:28.195760 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:28.195763 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:28.195816 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:28.195901 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:28.196338 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:28.196874 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:28.197702 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:28.197930 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:28.198034 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:28.200554 (+ 2520us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50239: BlockingRecv error: recv got EOF from 127.0.98.3:50239 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 27 12:41:28 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:41:28 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:41:28.838627 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:28.826543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42343 (local address 127.0.98.1:38547)
0927 12:41:28.826848 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:28.826861 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:28.826955 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:28.827249 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:28.827252 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:28.827456 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:28.827573 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:28.827580 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:28.828355 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:28.828357 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:28.829031 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:28.829040 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:28.829614 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:28.829622 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:28.829691 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:28.836541 (+ 6850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:28.836555 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:28.838507 (+ 1952us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":83,"threads_started":1}
W20260927 12:41:28.838837 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:41:28.826648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38547 (local address 127.0.98.62:42343)
0927 12:41:28.826902 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:28.826905 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:28.826918 (+ 13us) server_negotiation.cc:407] Connection header received
0927 12:41:28.827046 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:28.827049 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:28.827090 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:28.827174 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:28.827721 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:28.828243 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:28.829169 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:28.829447 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:28.829551 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:28.838680 (+ 9129us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38547: BlockingRecv error: recv got EOF from 127.0.98.1:38547 (error 108)
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":76,"threads_started":1}
Sep 27 12:41:29 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:41:29 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:41:29.020016 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:29.014765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42343 (local address 127.0.98.2:60655)
0927 12:41:29.015084 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:29.015100 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:29.015223 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:29.015502 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:29.015505 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:29.015701 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:29.015790 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:29.015795 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:29.016604 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:29.016607 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:29.017153 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:29.017160 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:29.017559 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:29.017562 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:29.017649 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:29.018205 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:29.018217 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:29.019889 (+ 1672us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":101,"threads_started":1}
W20260927 12:41:29.020159 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:41:29.014903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60655 (local address 127.0.98.62:42343)
0927 12:41:29.015054 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:29.015059 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:29.015194 (+ 135us) server_negotiation.cc:407] Connection header received
0927 12:41:29.015330 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:29.015333 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:29.015384 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:29.015480 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:29.015895 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:29.016499 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:29.017250 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:29.017469 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:29.017546 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:29.020044 (+ 2498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:60655: BlockingRecv error: recv got EOF from 127.0.98.2:60655 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 27 12:41:29 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:41:29 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:41:29.206485 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:29.201427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42343 (local address 127.0.98.3:49839)
0927 12:41:29.201705 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:29.201719 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:29.201846 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:29.202091 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:29.202093 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:29.202261 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:29.202351 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:29.202356 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:29.203081 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:29.203083 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:29.203643 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:29.203649 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:29.204049 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:29.204052 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:29.204109 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:29.204624 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:29.204642 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:29.206367 (+ 1725us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":79,"threads_started":1}
W20260927 12:41:29.206617 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:41:29.201499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49839 (local address 127.0.98.62:42343)
0927 12:41:29.201649 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:29.201653 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:29.201796 (+ 143us) server_negotiation.cc:407] Connection header received
0927 12:41:29.201932 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:29.201935 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:29.201989 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:29.202055 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:29.202457 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:29.202990 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:29.203731 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:29.203961 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:29.204036 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:29.206504 (+ 2468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:49839: BlockingRecv error: recv got EOF from 127.0.98.3:49839 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 27 12:41:29 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:41:29 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:41:29.845474 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:29.839510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42343 (local address 127.0.98.1:57159)
0927 12:41:29.839782 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:29.839795 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:29.839946 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:29.840191 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:29.840193 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:29.840397 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:29.840492 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:29.840497 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:29.841389 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:29.841393 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:29.842336 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:29.842347 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:29.842768 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:29.842770 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:29.842862 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:29.843396 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:29.843409 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:29.845305 (+ 1896us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":95,"threads_started":1}
W20260927 12:41:29.845558 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:41:29.839572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57159 (local address 127.0.98.62:42343)
0927 12:41:29.839795 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:29.839798 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:29.839907 (+ 109us) server_negotiation.cc:407] Connection header received
0927 12:41:29.840033 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:29.840036 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:29.840079 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:29.840162 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:29.840618 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:29.841231 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:29.842438 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:29.842667 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:29.842760 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:29.845434 (+ 2674us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:57159: BlockingRecv error: recv got EOF from 127.0.98.1:57159 (error 108)
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":77,"threads_started":1}
Sep 27 12:41:30 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:41:30 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:41:30.026212 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:30.020767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42343 (local address 127.0.98.2:35751)
0927 12:41:30.021039 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:30.021057 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:30.021197 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:30.021452 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:30.021455 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:30.021682 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:30.021771 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:30.021777 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:30.022548 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:30.022551 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:30.023105 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:30.023113 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:30.023551 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:30.023554 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:30.023646 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:30.024145 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:30.024156 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:30.026085 (+ 1929us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":84,"threads_started":1}
W20260927 12:41:30.026335 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:41:30.020825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35751 (local address 127.0.98.62:42343)
0927 12:41:30.021004 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:30.021009 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:30.021138 (+ 129us) server_negotiation.cc:407] Connection header received
0927 12:41:30.021291 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:30.021294 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:30.021345 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:30.021426 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:30.021890 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:30.022432 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:30.023221 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:30.023444 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:30.023550 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:30.026224 (+ 2674us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35751: BlockingRecv error: recv got EOF from 127.0.98.2:35751 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 27 12:41:30 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:41:30 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:41:30.212849 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:30.207397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42343 (local address 127.0.98.3:53369)
0927 12:41:30.207706 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:30.207719 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:30.207818 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:30.208079 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:30.208081 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:30.208256 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:30.208349 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:30.208354 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:30.209099 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:30.209101 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:30.209698 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:30.209706 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:30.210126 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:30.210128 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:30.210186 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:30.210749 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:30.210761 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:30.212683 (+ 1922us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":140,"threads_started":1}
W20260927 12:41:30.212916 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:41:30.207454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53369 (local address 127.0.98.62:42343)
0927 12:41:30.207621 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:30.207625 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:30.207833 (+ 208us) server_negotiation.cc:407] Connection header received
0927 12:41:30.207880 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:30.207882 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:30.207929 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:30.208045 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:30.208448 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:30.208963 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:30.209814 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:30.210028 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:30.210127 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:30.212807 (+ 2680us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:53369: BlockingRecv error: recv got EOF from 127.0.98.3:53369 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 27 12:41:30 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:41:30 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:41:30.851989 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:30.846387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42343 (local address 127.0.98.1:35669)
0927 12:41:30.846707 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:30.846723 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:30.846846 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:30.847169 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:30.847173 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:30.847412 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:30.847507 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:30.847513 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:30.848369 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:30.848375 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:30.848952 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:30.848960 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:30.849405 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:30.849409 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:30.849494 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:30.850004 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:30.850016 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:30.851810 (+ 1794us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":70,"threads_started":1}
W20260927 12:41:30.852063 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:41:30.846442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35669 (local address 127.0.98.62:42343)
0927 12:41:30.846808 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:30.846812 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:30.846832 (+ 20us) server_negotiation.cc:407] Connection header received
0927 12:41:30.846998 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:30.847000 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:30.847045 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:30.847146 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:30.847622 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:30.848265 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:30.849066 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:30.849298 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:30.849403 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:30.851935 (+ 2532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35669: BlockingRecv error: recv got EOF from 127.0.98.1:35669 (error 108)
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":160,"threads_started":1}
Sep 27 12:41:31 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:41:31 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:41:31.032517 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:31.027161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42343 (local address 127.0.98.2:48633)
0927 12:41:31.027432 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:31.027446 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:31.027549 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:31.027854 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:31.027857 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:31.028033 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:31.028127 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:31.028132 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:31.028926 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:31.028929 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:31.029532 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:31.029541 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:31.030028 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:31.030030 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:31.030092 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:31.030643 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:31.030655 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:31.032394 (+ 1739us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":99,"threads_started":1}
W20260927 12:41:31.032631 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:41:31.027316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48633 (local address 127.0.98.62:42343)
0927 12:41:31.027447 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:31.027451 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:31.027520 (+ 69us) server_negotiation.cc:407] Connection header received
0927 12:41:31.027693 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:31.027696 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:31.027744 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:31.027822 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:31.028247 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:31.028814 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:31.029692 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:31.029927 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:31.030018 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:31.032520 (+ 2502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48633: BlockingRecv error: recv got EOF from 127.0.98.2:48633 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 27 12:41:31 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:41:31 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:41:31.219249 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:31.213723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42343 (local address 127.0.98.3:48501)
0927 12:41:31.214018 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:31.214033 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:31.214172 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:31.214416 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:31.214418 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:31.214640 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:31.214732 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:31.214737 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:31.215444 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:31.215446 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:31.216010 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:31.216016 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:31.216556 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:31.216560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:31.216653 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:31.217242 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:31.217257 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:31.219126 (+ 1869us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":100,"threads_started":1}
W20260927 12:41:31.219444 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:41:31.213815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48501 (local address 127.0.98.62:42343)
0927 12:41:31.213989 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:31.213994 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:31.214142 (+ 148us) server_negotiation.cc:407] Connection header received
0927 12:41:31.214251 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:31.214253 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:31.214301 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:31.214373 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:31.214827 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:31.215358 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:31.216140 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:31.216430 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:31.216539 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:31.219304 (+ 2765us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:48501: BlockingRecv error: recv got EOF from 127.0.98.3:48501 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 27 12:41:31 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:41:31 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:41:31.858238 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:31.852690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42343 (local address 127.0.98.1:38279)
0927 12:41:31.852947 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:31.852960 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:31.853091 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:31.853328 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:31.853330 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:31.853518 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:31.853681 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:31.853687 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:31.854594 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:31.854599 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:31.855162 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:31.855170 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:31.855628 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:31.855631 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:31.855715 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:31.856193 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:31.856206 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:31.858065 (+ 1859us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":89,"threads_started":1}
W20260927 12:41:31.858310 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:41:31.852763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38279 (local address 127.0.98.62:42343)
0927 12:41:31.852975 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:31.852978 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:31.853046 (+ 68us) server_negotiation.cc:407] Connection header received
0927 12:41:31.853179 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:31.853182 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:31.853224 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:31.853294 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:31.853790 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:31.854463 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:31.855275 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:31.855514 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:31.855619 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:31.858188 (+ 2569us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38279: BlockingRecv error: recv got EOF from 127.0.98.1:38279 (error 108)
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":82,"threads_started":1}
Sep 27 12:41:32 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:41:32 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:41:32.038976 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:32.033499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42343 (local address 127.0.98.2:44735)
0927 12:41:32.033781 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:32.033794 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:32.033895 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:32.034113 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:32.034115 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:32.034300 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:32.034390 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:32.034395 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:32.035130 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:32.035133 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:32.035943 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:32.035970 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:32.036511 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:32.036514 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:32.036584 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:32.037043 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:32.037053 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:32.038864 (+ 1811us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":109,"threads_started":1}
W20260927 12:41:32.039157 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:41:32.033619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44735 (local address 127.0.98.62:42343)
0927 12:41:32.033749 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:32.033752 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:32.033846 (+ 94us) server_negotiation.cc:407] Connection header received
0927 12:41:32.033961 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:32.033963 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:32.034006 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:32.034084 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:32.034474 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:32.035038 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:32.036058 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:32.036391 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:32.036491 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:32.039008 (+ 2517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44735: BlockingRecv error: recv got EOF from 127.0.98.2:44735 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 27 12:41:32 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:41:32 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:41:32.225831 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:32.220108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42343 (local address 127.0.98.3:35511)
0927 12:41:32.220565 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:32.220582 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:32.220708 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:32.221010 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:32.221013 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:32.221229 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:32.221323 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:32.221329 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:32.222167 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:32.222173 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:32.222735 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:32.222743 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:32.223214 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:32.223217 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:32.223303 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:32.223844 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:32.223867 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:32.225718 (+ 1851us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":130,"threads_started":1}
W20260927 12:41:32.225984 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:41:32.220278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35511 (local address 127.0.98.62:42343)
0927 12:41:32.220567 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:32.220573 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:32.220649 (+ 76us) server_negotiation.cc:407] Connection header received
0927 12:41:32.220833 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:32.220836 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:32.220895 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:32.220985 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:32.221443 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:32.222060 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:32.222861 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:32.223105 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:32.223211 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:32.225854 (+ 2643us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:35511: BlockingRecv error: recv got EOF from 127.0.98.3:35511 (error 108)
Metrics: {"server-negotiator.queue_time_us":189}
Sep 27 12:41:32 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:41:32 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:41:32.864624 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:32.859088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42343 (local address 127.0.98.1:41567)
0927 12:41:32.859412 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:32.859425 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:32.859540 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:32.859773 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:32.859775 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:32.859978 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:32.860107 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:32.860113 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:32.860961 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:32.860968 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:32.861616 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:32.861631 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:32.862037 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:32.862039 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:32.862099 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:32.862660 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:32.862671 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:32.864507 (+ 1836us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":85,"threads_started":1}
W20260927 12:41:32.864859 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:41:32.859133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41567 (local address 127.0.98.62:42343)
0927 12:41:32.859426 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:32.859430 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:32.859511 (+ 81us) server_negotiation.cc:407] Connection header received
0927 12:41:32.859615 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:32.859617 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:32.859661 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:32.859739 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:32.860238 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:32.860844 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:32.861721 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:32.861945 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:32.862026 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:32.864716 (+ 2690us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:41567: BlockingRecv error: recv got EOF from 127.0.98.1:41567 (error 108)
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":88,"threads_started":1}
Sep 27 12:41:33 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:41:33 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:41:33.045400 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:33.039751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42343 (local address 127.0.98.2:58903)
0927 12:41:33.040055 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:33.040068 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:33.040201 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:33.040481 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:33.040487 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:33.040677 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:33.040826 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:33.040837 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:33.041851 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:33.041855 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:33.042442 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:33.042450 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:33.042850 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:33.042853 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:33.042975 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:33.043460 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:33.043472 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:33.045289 (+ 1817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":108,"threads_started":1}
W20260927 12:41:33.045517 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:41:33.039836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58903 (local address 127.0.98.62:42343)
0927 12:41:33.039970 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:33.039974 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:33.040169 (+ 195us) server_negotiation.cc:407] Connection header received
0927 12:41:33.040290 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:33.040293 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:33.040352 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:33.040434 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:33.040934 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:33.041737 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:33.042535 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:33.042753 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:33.042853 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:33.045409 (+ 2556us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58903: BlockingRecv error: recv got EOF from 127.0.98.2:58903 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 27 12:41:33 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:41:33 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:41:33.232378 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:33.226628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42343 (local address 127.0.98.3:53365)
0927 12:41:33.226994 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:33.227007 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:33.227159 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:33.227478 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:33.227482 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:33.227707 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:33.227847 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:33.227854 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:33.228623 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:33.228627 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:33.229191 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:33.229199 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:33.229691 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:33.229695 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:33.229780 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:33.230256 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:33.230268 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:33.232204 (+ 1936us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":75,"threads_started":1}
W20260927 12:41:33.232437 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:41:33.226758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53365 (local address 127.0.98.62:42343)
0927 12:41:33.226901 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:33.226905 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:33.227082 (+ 177us) server_negotiation.cc:407] Connection header received
0927 12:41:33.227260 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:33.227263 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:33.227339 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:33.227422 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:33.227965 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:33.228504 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:33.229310 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:33.229551 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:33.229688 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:33.232329 (+ 2641us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:53365: BlockingRecv error: recv got EOF from 127.0.98.3:53365 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 27 12:41:33 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:41:33 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:41:33.875416 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:33.865457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42343 (local address 127.0.98.1:37991)
0927 12:41:33.865756 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:33.865779 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:33.865881 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:33.866209 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:33.866212 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:33.866451 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:33.866597 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:33.866609 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:33.867617 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:33.867620 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:33.868207 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:33.868217 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:33.868678 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:33.868682 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:33.868753 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:33.873437 (+ 4684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:33.873450 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:33.875301 (+ 1851us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":149,"threads_started":1}
W20260927 12:41:33.875619 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:41:33.865654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37991 (local address 127.0.98.62:42343)
0927 12:41:33.865922 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:33.865927 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:33.865944 (+ 17us) server_negotiation.cc:407] Connection header received
0927 12:41:33.865993 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:33.865996 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:33.866049 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:33.866137 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:33.866737 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:33.867500 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:33.868335 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:33.868554 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:33.868634 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:33.875452 (+ 6818us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:37991: BlockingRecv error: recv got EOF from 127.0.98.1:37991 (error 108)
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":90,"threads_started":1}
Sep 27 12:41:34 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:41:34 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:41:34.051769 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:34.046177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42343 (local address 127.0.98.2:36853)
0927 12:41:34.046466 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:34.046482 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:34.046612 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:34.046827 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:34.046829 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:34.047011 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:34.047103 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:34.047108 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:34.048040 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:34.048042 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:34.048739 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:34.048748 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:34.049141 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:34.049143 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:34.049200 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:34.049751 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:34.049763 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:34.051577 (+ 1814us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":94,"threads_started":1}
W20260927 12:41:34.051968 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:41:34.046227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36853 (local address 127.0.98.62:42343)
0927 12:41:34.046378 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:34.046381 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:34.046610 (+ 229us) server_negotiation.cc:407] Connection header received
0927 12:41:34.046677 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:34.046679 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:34.046726 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:34.046799 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:34.047256 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:34.047946 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:34.048838 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:34.049051 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:34.049128 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:34.051755 (+ 2627us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:36853: BlockingRecv error: recv got EOF from 127.0.98.2:36853 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 27 12:41:34 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:41:34 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:41:34.238970 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:34.233273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42343 (local address 127.0.98.3:46239)
0927 12:41:34.233542 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:34.233555 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:34.233666 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:34.233978 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:34.233980 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:34.234186 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:34.234283 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:34.234288 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:34.235135 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:34.235142 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:34.235841 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:34.235850 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:34.236428 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:34.236434 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:34.236516 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:34.237047 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:34.237059 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:34.238839 (+ 1780us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":68,"threads_started":1}
W20260927 12:41:34.239238 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:41:34.233377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46239 (local address 127.0.98.62:42343)
0927 12:41:34.233559 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:34.233603 (+ 44us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:34.233623 (+ 20us) server_negotiation.cc:407] Connection header received
0927 12:41:34.233792 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:34.233796 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:34.233853 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:34.233927 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:34.234402 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:34.235010 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:34.235963 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:34.236286 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:34.236419 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:34.238990 (+ 2571us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46239: BlockingRecv error: recv got EOF from 127.0.98.3:46239 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Sep 27 12:41:34 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:41:34 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:41:34.882547 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:34.876214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42343 (local address 127.0.98.1:34771)
0927 12:41:34.876581 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:34.876597 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:34.876742 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:34.877011 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:34.877014 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:34.877216 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:34.877304 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:34.877309 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:34.878142 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:34.878149 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:34.878717 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:34.878726 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:34.879430 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:34.879437 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:34.879532 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:34.880104 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:34.880120 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:34.882363 (+ 2243us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":87,"threads_started":1}
W20260927 12:41:34.882614 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:41:34.876354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34771 (local address 127.0.98.62:42343)
0927 12:41:34.876583 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:34.876588 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:34.876680 (+ 92us) server_negotiation.cc:407] Connection header received
0927 12:41:34.876838 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:34.876841 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:34.876890 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:34.876992 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:34.877418 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:34.878032 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:34.878837 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:34.879160 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:34.879275 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:34.882494 (+ 3219us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:34771: BlockingRecv error: recv got EOF from 127.0.98.1:34771 (error 108)
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":57,"threads_started":1}
Sep 27 12:41:35 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:41:35 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:41:35.057869 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:35.052500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42343 (local address 127.0.98.2:53525)
0927 12:41:35.052743 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:35.052764 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:35.052898 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:35.053196 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:35.053199 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:35.053449 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:35.053633 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:35.053641 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:35.054445 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:35.054447 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:35.055041 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:35.055050 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:35.055439 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:35.055441 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:35.055502 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:35.055971 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:35.055982 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:35.057759 (+ 1777us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":84,"threads_started":1}
W20260927 12:41:35.058121 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:41:35.052611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53525 (local address 127.0.98.62:42343)
0927 12:41:35.052726 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:35.052740 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:35.052851 (+ 111us) server_negotiation.cc:407] Connection header received
0927 12:41:35.053003 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:35.053006 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:35.053068 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:35.053148 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:35.053738 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:35.054359 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:35.055139 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:35.055346 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:35.055423 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:35.057883 (+ 2460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53525: BlockingRecv error: recv got EOF from 127.0.98.2:53525 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 27 12:41:35 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:41:35 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:41:35.245368 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:35.239689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42343 (local address 127.0.98.3:39081)
0927 12:41:35.239984 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:35.239997 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:35.240106 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:35.240369 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:35.240372 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:35.240568 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:35.240664 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:35.240669 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:35.241499 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:35.241502 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:35.242250 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:35.242260 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:35.242702 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:35.242705 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:35.242768 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:35.243399 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:35.243413 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:35.245245 (+ 1832us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":108,"threads_started":1}
W20260927 12:41:35.245522 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:41:35.239761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39081 (local address 127.0.98.62:42343)
0927 12:41:35.239903 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:35.239908 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:35.240079 (+ 171us) server_negotiation.cc:407] Connection header received
0927 12:41:35.240183 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:35.240185 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:35.240234 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:35.240307 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:35.240757 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:35.241390 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:35.242359 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:35.242575 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:35.242661 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:35.245379 (+ 2718us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:39081: BlockingRecv error: recv got EOF from 127.0.98.3:39081 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 27 12:41:35 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:41:35 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:41:35.888970 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:35.883463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42343 (local address 127.0.98.1:45899)
0927 12:41:35.883778 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:35.883791 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:35.883880 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:35.884171 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:35.884173 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:35.884447 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:35.884548 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:35.884555 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:35.885453 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:35.885458 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:35.886085 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:35.886093 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:35.886540 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:35.886544 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:35.886627 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:35.887113 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:35.887124 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:35.888808 (+ 1684us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":90,"threads_started":1}
W20260927 12:41:35.889043 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:41:35.883542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45899 (local address 127.0.98.62:42343)
0927 12:41:35.883828 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:35.883832 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:35.883845 (+ 13us) server_negotiation.cc:407] Connection header received
0927 12:41:35.883982 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:35.883988 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:35.884043 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:35.884140 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:35.884655 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:35.885334 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:35.886212 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:35.886436 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:35.886538 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:35.888927 (+ 2389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45899: BlockingRecv error: recv got EOF from 127.0.98.1:45899 (error 108)
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":116,"threads_started":1}
Sep 27 12:41:36 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:41:36 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:41:36.064332 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:36.058650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42343 (local address 127.0.98.2:48537)
0927 12:41:36.058975 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:36.058989 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:36.059125 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:36.059524 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:36.059530 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:36.059778 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:36.059908 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:36.059914 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:36.060765 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:36.060770 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:36.061336 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:36.061344 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:36.061853 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:36.061857 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:36.061943 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:36.062406 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:36.062418 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:36.064217 (+ 1799us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":109,"threads_started":1}
W20260927 12:41:36.064497 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:41:36.058741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48537 (local address 127.0.98.62:42343)
0927 12:41:36.058893 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:36.058900 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:36.059231 (+ 331us) server_negotiation.cc:407] Connection header received
0927 12:41:36.059271 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:36.059274 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:36.059327 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:36.059416 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:36.060016 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:36.060655 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:36.061453 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:36.061744 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:36.061849 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:36.064349 (+ 2500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48537: BlockingRecv error: recv got EOF from 127.0.98.2:48537 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 27 12:41:36 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:41:36 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:41:36.251248 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:36.246150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42343 (local address 127.0.98.3:37353)
0927 12:41:36.246388 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:36.246402 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:36.246481 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:36.246707 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:36.246709 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:36.246880 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:36.246979 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:36.246984 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:36.247750 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:36.247754 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:36.248312 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:36.248319 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:36.248708 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:36.248711 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:36.248772 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:36.249280 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:36.249291 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:36.251086 (+ 1795us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":69,"threads_started":1}
W20260927 12:41:36.251350 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:41:36.246281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37353 (local address 127.0.98.62:42343)
0927 12:41:36.246428 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:36.246432 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:36.246446 (+ 14us) server_negotiation.cc:407] Connection header received
0927 12:41:36.246559 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:36.246561 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:36.246622 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:36.246690 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:36.247065 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:36.247614 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:36.248406 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:36.248617 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:36.248694 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:36.251240 (+ 2546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:37353: BlockingRecv error: recv got EOF from 127.0.98.3:37353 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 27 12:41:36 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:41:36 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:41:36.894953 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:36.889723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42343 (local address 127.0.98.1:45327)
0927 12:41:36.890018 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:36.890031 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:36.890119 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:36.890363 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:36.890365 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:36.890535 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:36.890627 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:36.890631 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:36.891348 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:36.891359 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:36.891898 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:36.891905 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:36.892302 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:36.892305 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:36.892374 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:36.892899 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:36.892910 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:36.894833 (+ 1923us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":80,"threads_started":1}
W20260927 12:41:36.895148 9591 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:41:36.889794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45327 (local address 127.0.98.62:42343)
0927 12:41:36.890057 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:36.890061 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:36.890073 (+ 12us) server_negotiation.cc:407] Connection header received
0927 12:41:36.890200 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:36.890202 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:36.890243 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:36.890321 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:36.890715 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:36.891255 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:36.891990 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:36.892203 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:36.892294 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:36.894957 (+ 2663us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45327: BlockingRecv error: recv got EOF from 127.0.98.1:45327 (error 108)
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":114,"threads_started":1}
Sep 27 12:41:37 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:41:37 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:41:37.070030 9592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:37.065119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42343 (local address 127.0.98.2:37513)
0927 12:41:37.065396 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:37.065410 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:37.065505 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:37.065728 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:37.065730 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:37.065904 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:37.065996 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:37.066000 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:37.066703 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:37.066706 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:37.067245 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:37.067252 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:37.067638 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:37.067640 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:37.067702 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:37.068157 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:37.068168 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:37.069919 (+ 1751us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":84,"threads_started":1}
W20260927 12:41:37.070174 9591 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:41:37.065195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37513 (local address 127.0.98.62:42343)
0927 12:41:37.065395 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:37.065398 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:37.065478 (+ 80us) server_negotiation.cc:407] Connection header received
0927 12:41:37.065605 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:37.065607 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:37.065648 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:37.065717 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:37.066090 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:37.066613 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:37.067341 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:37.067548 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:37.067621 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:37.070061 (+ 2440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37513: BlockingRecv error: recv got EOF from 127.0.98.2:37513 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Sep 27 12:41:37 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:41:37 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:41:37.257427 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:37.251981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42343 (local address 127.0.98.3:33923)
0927 12:41:37.252316 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:37.252328 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:37.252461 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:37.252731 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:37.252733 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:37.252941 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:37.253029 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:37.253034 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:37.253852 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:37.253856 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:37.254424 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:37.254432 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:37.254913 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:37.254917 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:37.255009 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:37.255552 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:37.255563 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:37.257313 (+ 1750us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":90,"threads_started":1}
W20260927 12:41:37.257560 9591 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:41:37.252037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33923 (local address 127.0.98.62:42343)
0927 12:41:37.252193 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:37.252200 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:37.252420 (+ 220us) server_negotiation.cc:407] Connection header received
0927 12:41:37.252568 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:37.252571 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:37.252625 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:37.252710 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:37.253142 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:37.253745 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:37.254546 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:37.254791 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:37.254902 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:37.257445 (+ 2543us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:33923: BlockingRecv error: recv got EOF from 127.0.98.3:33923 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 27 12:41:37 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:41:37 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:41:37.900744 9594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:37.895729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42343 (local address 127.0.98.1:52165)
0927 12:41:37.896009 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:37.896038 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:37.896134 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:37.896349 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:37.896352 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:37.896530 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:37.896619 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:37.896624 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:37.897307 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:37.897309 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:37.897910 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:37.897917 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:37.898322 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:37.898325 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:37.898384 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:37.898874 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:37.898885 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:37.900632 (+ 1747us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":69,"threads_started":1}
W20260927 12:41:37.900924 9595 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:41:37.895857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52165 (local address 127.0.98.62:42343)
0927 12:41:37.896098 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:37.896101 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:37.896114 (+ 13us) server_negotiation.cc:407] Connection header received
0927 12:41:37.896200 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:37.896202 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:37.896240 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:37.896313 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:37.896713 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:37.897221 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:37.898007 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:37.898226 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:37.898304 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:37.900758 (+ 2454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:52165: BlockingRecv error: recv got EOF from 127.0.98.1:52165 (error 108)
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":85,"threads_started":1}
Sep 27 12:41:38 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:41:38 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:41:38.076381 9596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:38.070886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42343 (local address 127.0.98.2:37813)
0927 12:41:38.071178 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:38.071191 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:38.071292 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:38.071582 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:38.071585 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:38.071788 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:38.071913 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:38.071920 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:38.072661 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:38.072664 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:38.073292 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:38.073301 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:38.073826 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:38.073830 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:38.073900 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:38.074383 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:38.074395 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:38.076212 (+ 1817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":118,"threads_started":1}
W20260927 12:41:38.076444 9595 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:41:38.070938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37813 (local address 127.0.98.62:42343)
0927 12:41:38.071084 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:38.071102 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:38.071242 (+ 140us) server_negotiation.cc:407] Connection header received
0927 12:41:38.071359 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:38.071360 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:38.071402 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:38.071469 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:38.072029 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:38.072538 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:38.073444 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:38.073722 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:38.073798 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:38.076339 (+ 2541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37813: BlockingRecv error: recv got EOF from 127.0.98.2:37813 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 27 12:41:38 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:41:38 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:41:38.263412 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:38.258257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42343 (local address 127.0.98.3:59053)
0927 12:41:38.258551 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:38.258565 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:38.258666 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:38.258865 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:38.258868 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:38.259042 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:38.259132 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:38.259137 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:38.259846 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:38.259848 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:38.260464 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:38.260472 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:38.260906 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:38.260909 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:38.260972 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:38.261479 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:38.261519 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:38.263300 (+ 1781us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":112,"threads_started":1}
W20260927 12:41:38.263537 9595 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:41:38.258309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59053 (local address 127.0.98.62:42343)
0927 12:41:38.258440 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:38.258445 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:38.258618 (+ 173us) server_negotiation.cc:407] Connection header received
0927 12:41:38.258729 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:38.258731 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:38.258772 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:38.258837 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:38.259224 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:38.259721 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:38.260566 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:38.260790 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:38.260867 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:38.263425 (+ 2558us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:59053: BlockingRecv error: recv got EOF from 127.0.98.3:59053 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 27 12:41:38 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:41:38 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:41:38.910666 9598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:38.901672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42343 (local address 127.0.98.1:57077)
0927 12:41:38.901953 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:38.901966 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:38.902072 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:38.902315 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:38.902317 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:38.902489 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:38.902607 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:38.902614 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:38.903335 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:38.903338 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:38.903914 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:38.903922 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:38.904486 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:38.904489 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:38.904579 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:38.908703 (+ 4124us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:38.908715 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:38.910461 (+ 1746us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":84,"threads_started":1}
W20260927 12:41:38.910703 9599 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:41:38.901725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57077 (local address 127.0.98.62:42343)
0927 12:41:38.901967 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:38.901970 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:38.902051 (+ 81us) server_negotiation.cc:407] Connection header received
0927 12:41:38.902156 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:38.902158 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:38.902202 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:38.902279 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:38.902713 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:38.903238 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:38.904051 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:38.904365 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:38.904482 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:38.910586 (+ 6104us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:57077: BlockingRecv error: recv got EOF from 127.0.98.1:57077 (error 108)
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":59,"threads_started":1}
Sep 27 12:41:39 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:41:39 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:41:39.082938 9600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:39.077329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42343 (local address 127.0.98.2:51827)
0927 12:41:39.077679 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:39.077693 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:39.077793 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:39.078026 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:39.078029 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:39.078244 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:39.078341 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:39.078347 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:39.079206 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:39.079211 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:39.079765 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:39.079772 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:39.080231 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:39.080234 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:39.080317 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:39.080795 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:39.080807 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:39.082771 (+ 1964us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":94,"threads_started":1}
W20260927 12:41:39.083001 9599 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:41:39.077594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51827 (local address 127.0.98.62:42343)
0927 12:41:39.077742 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:39.077746 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:39.077760 (+ 14us) server_negotiation.cc:407] Connection header received
0927 12:41:39.077862 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:39.077864 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:39.077904 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:39.077973 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:39.078469 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:39.079094 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:39.079882 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:39.080113 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:39.080229 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:39.082899 (+ 2670us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:51827: BlockingRecv error: recv got EOF from 127.0.98.2:51827 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 27 12:41:39 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:41:39 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:41:39.269851 9601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:39.264339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42343 (local address 127.0.98.3:41681)
0927 12:41:39.264665 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:39.264681 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:39.264816 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:39.265157 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:39.265160 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:39.265389 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:39.265522 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:39.265530 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:39.266428 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:39.266430 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:39.266984 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:39.266991 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:39.267406 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:39.267409 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:39.267466 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:39.267951 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:39.267961 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:39.269715 (+ 1754us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":78,"threads_started":1}
W20260927 12:41:39.269968 9599 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:41:39.264478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41681 (local address 127.0.98.62:42343)
0927 12:41:39.264630 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:39.264635 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:39.264751 (+ 116us) server_negotiation.cc:407] Connection header received
0927 12:41:39.264918 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:39.264921 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:39.265001 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:39.265097 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:39.265679 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:39.266329 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:39.267077 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:39.267300 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:39.267374 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:39.269852 (+ 2478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:41681: BlockingRecv error: recv got EOF from 127.0.98.3:41681 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 27 12:41:39 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:41:39 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:41:39.917109 9602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:39.911456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42343 (local address 127.0.98.1:57427)
0927 12:41:39.911774 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:39.911788 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:39.911878 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:39.912114 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:39.912116 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:39.912294 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:39.912390 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:39.912395 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:39.913106 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:39.913109 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:39.913762 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:39.913770 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:39.914188 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:39.914190 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:39.914256 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:39.914832 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:39.914844 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:39.916930 (+ 2086us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":89,"threads_started":1}
W20260927 12:41:39.917258 9603 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:41:39.911573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57427 (local address 127.0.98.62:42343)
0927 12:41:39.911858 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:39.911862 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:39.911875 (+ 13us) server_negotiation.cc:407] Connection header received
0927 12:41:39.911959 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:39.911961 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:39.912002 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:39.912077 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:39.912488 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:39.913014 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:39.913868 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:39.914088 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:39.914165 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:39.917106 (+ 2941us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:57427: BlockingRecv error: recv got EOF from 127.0.98.1:57427 (error 108)
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":60,"threads_started":1}
Sep 27 12:41:40 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:41:40 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:41:40.088842 9604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:40.083697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42343 (local address 127.0.98.2:38137)
0927 12:41:40.083974 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:40.083987 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:40.084084 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:40.084306 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:40.084308 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:40.084489 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:40.084581 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:40.084587 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:40.085273 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:40.085275 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:40.085851 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:40.085858 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:40.086251 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:40.086253 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:40.086320 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:40.086792 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:40.086803 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:40.088710 (+ 1907us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":114,"threads_started":1}
W20260927 12:41:40.088989 9603 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:41:40.083745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38137 (local address 127.0.98.62:42343)
0927 12:41:40.083896 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:40.083901 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:40.084058 (+ 157us) server_negotiation.cc:407] Connection header received
0927 12:41:40.084161 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:40.084163 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:40.084207 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:40.084291 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:40.084675 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:40.085185 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:40.085946 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:40.086156 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:40.086233 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:40.088848 (+ 2615us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:38137: BlockingRecv error: recv got EOF from 127.0.98.2:38137 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 27 12:41:40 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:41:40 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:41:40.276053 9605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:40.270669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42343 (local address 127.0.98.3:44111)
0927 12:41:40.270940 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:40.270953 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:40.271053 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:40.271280 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:40.271282 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:40.271466 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:40.271555 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:40.271560 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:40.272281 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:40.272284 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:40.272820 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:40.272826 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:40.273353 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:40.273356 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:40.273450 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:40.274081 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:40.274095 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:40.275932 (+ 1837us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":94,"threads_started":1}
W20260927 12:41:40.276185 9603 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:41:40.270765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44111 (local address 127.0.98.62:42343)
0927 12:41:40.270900 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:40.270903 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:40.271023 (+ 120us) server_negotiation.cc:407] Connection header received
0927 12:41:40.271127 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:40.271129 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:40.271172 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:40.271246 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:40.271652 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:40.272191 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:40.272916 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:40.273197 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:40.273307 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:40.276070 (+ 2763us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:44111: BlockingRecv error: recv got EOF from 127.0.98.3:44111 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 27 12:41:40 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:41:40 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:41:40.923218 9606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:40.918036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42343 (local address 127.0.98.1:55923)
0927 12:41:40.918280 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:40.918293 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:40.918372 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:40.918609 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:40.918611 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:40.918777 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:40.918868 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:40.918872 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:40.919676 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:40.919679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:40.920289 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:40.920296 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:40.920727 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:40.920730 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:40.920786 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:40.921260 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:40.921271 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:40.923050 (+ 1779us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":84,"threads_started":1}
W20260927 12:41:40.923292 9607 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:41:40.918088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55923 (local address 127.0.98.62:42343)
0927 12:41:40.918351 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:40.918354 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:40.918368 (+ 14us) server_negotiation.cc:407] Connection header received
0927 12:41:40.918440 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:40.918442 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:40.918482 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:40.918579 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:40.918965 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:40.919547 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:40.920386 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:40.920618 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:40.920711 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:40.923167 (+ 2456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:55923: BlockingRecv error: recv got EOF from 127.0.98.1:55923 (error 108)
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":76,"threads_started":1}
Sep 27 12:41:41 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 27 12:41:41 dist-test-slave-bnc1 krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260927 12:41:41.094754 9608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:41.089676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42343 (local address 127.0.98.2:35519)
0927 12:41:41.089948 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:41.089960 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:41.090067 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:41.090274 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:41.090276 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:41.090446 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:41.090552 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:41.090557 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:41.091250 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:41.091252 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:41.091824 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:41.091831 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:41.092262 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:41.092265 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:41.092327 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:41.092801 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:41.092813 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:41.094632 (+ 1819us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42343: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":112,"threads_started":1}
W20260927 12:41:41.094874 9607 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:41:41.089772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35519 (local address 127.0.98.62:42343)
0927 12:41:41.089900 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:41.089903 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:41.090047 (+ 144us) server_negotiation.cc:407] Connection header received
0927 12:41:41.090131 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:41.090133 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:41.090171 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:41.090239 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:41.090646 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:41.091161 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:41.091940 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:41.092159 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:41.092232 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:41.094762 (+ 2530us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35519: BlockingRecv error: recv got EOF from 127.0.98.2:35519 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 12:41:41.168778 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 9132
I20260927 12:41:41.174425 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 9284
I20260927 12:41:41.179905 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 9416
I20260927 12:41:41.185396 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 9062
2026-09-27T12:41:41Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17926 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-27T12:41:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:41:41Z Disabled control of system clock
I20260927 12:41:41.215194 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512840836255-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512840836255-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512840836255-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38659
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38659
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 12:41:41.327078 9612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:41:41.327338 9612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:41:41.327373 9612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:41:41.331337 9612 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:41:41.331388 9612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:41:41.331403 9612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:41:41.331420 9612 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:41:41.331481 9612 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:41:41.336143 9612 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512840836255-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512840836255-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38659
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9612
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512840836255-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:41:41.337165 9612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:41:41.338563 9612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:41:41.347167 9618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:41.347147 9620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:41.347148 9617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:41.347255 9612 server_base.cc:1034] running on GCE node
I20260927 12:41:41.347966 9612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:41:41.349052 9612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:41:41.350204 9612 hybrid_clock.cc:648] HybridClock initialized: now 1790512901350188 us; error 31 us; skew 500 ppm
I20260927 12:41:41.351962 9612 webserver.cc:460] Webserver started at http://127.0.98.62:33129/ using document root <none> and password file <none>
I20260927 12:41:41.352489 9612 fs_manager.cc:359] Metadata directory not provided
I20260927 12:41:41.352550 9612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:41:41.352705 9612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:41:41.354696 9612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512840836255-392-0/minicluster-data/master-0/data/instance:
uuid: "1becf9c54e034bf2a4716a7209ed75a7"
format_stamp: "Formatted at 2026-09-27 12:41:41 on dist-test-slave-bnc1"
server_key: "b9a58c89797ef6873bd85df334e5b4a9"
server_key_iv: "450c56de1b1866b85016744e9d3beeaa"
server_key_version: "encryptionkey@0"
I20260927 12:41:41.355258 9612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512840836255-392-0/minicluster-data/master-0/wal/instance:
uuid: "1becf9c54e034bf2a4716a7209ed75a7"
format_stamp: "Formatted at 2026-09-27 12:41:41 on dist-test-slave-bnc1"
server_key: "b9a58c89797ef6873bd85df334e5b4a9"
server_key_iv: "450c56de1b1866b85016744e9d3beeaa"
server_key_version: "encryptionkey@0"
I20260927 12:41:41.358729 9612 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 12:41:41.361166 9626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:41.362468 9612 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 12:41:41.362565 9612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512840836255-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512840836255-392-0/minicluster-data/master-0/wal
uuid: "1becf9c54e034bf2a4716a7209ed75a7"
format_stamp: "Formatted at 2026-09-27 12:41:41 on dist-test-slave-bnc1"
server_key: "b9a58c89797ef6873bd85df334e5b4a9"
server_key_iv: "450c56de1b1866b85016744e9d3beeaa"
server_key_version: "encryptionkey@0"
I20260927 12:41:41.362651 9612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512840836255-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512840836255-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512840836255-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:41:41.383599 9612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:41:41.384277 9612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:41:41.384424 9612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:41:41.391350 9612 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38659
I20260927 12:41:41.391355 9678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38659 every 8 connection(s)
I20260927 12:41:41.392302 9612 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:41:41.395408 9679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:41:41.395718 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 9612
I20260927 12:41:41.395824 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512840836255-392-0/minicluster-data/master-0/wal/instance
I20260927 12:41:41.396118 392 external_mini_cluster.cc:1428] Setting key 938fa6a35354dcad11f277d91ecf9e83
I20260927 12:41:41.402041 9679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1becf9c54e034bf2a4716a7209ed75a7: Bootstrap starting.
I20260927 12:41:41.404218 9679 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1becf9c54e034bf2a4716a7209ed75a7: Neither blocks nor log segments found. Creating new log.
I20260927 12:41:41.404958 9679 log.cc:824] T 00000000000000000000000000000000 P 1becf9c54e034bf2a4716a7209ed75a7: Log is configured to *not* fsync() on all Append() calls
I20260927 12:41:41.405467 9682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:41.397667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43292 (local address 127.0.98.62:38659)
0927 12:41:41.398291 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:41.398305 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:41.398338 (+ 33us) server_negotiation.cc:407] Connection header received
0927 12:41:41.398993 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:41.399010 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:41.399387 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:41.399726 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:41.400136 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:41.401072 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:41.402017 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:41.402572 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:41.402733 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:41.403594 (+ 861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:41.403599 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:41.403610 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:41:41.403978 (+ 368us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:41.404041 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:41:41.404061 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:41.404167 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:41.404285 (+ 118us) server_negotiation.cc:299] Negotiation successful
0927 12:41:41.404651 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":142,"threads_started":1}
I20260927 12:41:41.407064 9679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1becf9c54e034bf2a4716a7209ed75a7: No bootstrap required, opened a new log
I20260927 12:41:41.410507 9679 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1becf9c54e034bf2a4716a7209ed75a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1becf9c54e034bf2a4716a7209ed75a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38659 } }
I20260927 12:41:41.410773 9679 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1becf9c54e034bf2a4716a7209ed75a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:41:41.410848 9679 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1becf9c54e034bf2a4716a7209ed75a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1becf9c54e034bf2a4716a7209ed75a7, State: Initialized, Role: FOLLOWER
I20260927 12:41:41.411307 9679 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1becf9c54e034bf2a4716a7209ed75a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1becf9c54e034bf2a4716a7209ed75a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38659 } }
I20260927 12:41:41.411453 9679 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1becf9c54e034bf2a4716a7209ed75a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:41:41.411532 9679 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1becf9c54e034bf2a4716a7209ed75a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:41:41.411650 9679 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1becf9c54e034bf2a4716a7209ed75a7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:41:41.412513 9679 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1becf9c54e034bf2a4716a7209ed75a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1becf9c54e034bf2a4716a7209ed75a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38659 } }
I20260927 12:41:41.412846 9679 leader_election.cc:302] T 00000000000000000000000000000000 P 1becf9c54e034bf2a4716a7209ed75a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1becf9c54e034bf2a4716a7209ed75a7; no voters:
I20260927 12:41:41.413189 9679 leader_election.cc:288] T 00000000000000000000000000000000 P 1becf9c54e034bf2a4716a7209ed75a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:41:41.413321 9684 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1becf9c54e034bf2a4716a7209ed75a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:41:41.413520 9684 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1becf9c54e034bf2a4716a7209ed75a7 [term 1 LEADER]: Becoming Leader. State: Replica: 1becf9c54e034bf2a4716a7209ed75a7, State: Running, Role: LEADER
I20260927 12:41:41.413849 9684 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1becf9c54e034bf2a4716a7209ed75a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1becf9c54e034bf2a4716a7209ed75a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38659 } }
I20260927 12:41:41.414417 9679 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1becf9c54e034bf2a4716a7209ed75a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:41:41.415465 9685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1becf9c54e034bf2a4716a7209ed75a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1becf9c54e034bf2a4716a7209ed75a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1becf9c54e034bf2a4716a7209ed75a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38659 } } }
I20260927 12:41:41.415478 9686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1becf9c54e034bf2a4716a7209ed75a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1becf9c54e034bf2a4716a7209ed75a7. Latest consensus state: current_term: 1 leader_uuid: "1becf9c54e034bf2a4716a7209ed75a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1becf9c54e034bf2a4716a7209ed75a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38659 } } }
I20260927 12:41:41.415627 9685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1becf9c54e034bf2a4716a7209ed75a7 [sys.catalog]: This master's current role is: LEADER
I20260927 12:41:41.415669 9686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1becf9c54e034bf2a4716a7209ed75a7 [sys.catalog]: This master's current role is: LEADER
I20260927 12:41:41.416045 9693 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:41:41.419159 9693 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:41:41.424623 9693 catalog_manager.cc:1347] Generated new cluster ID: 92e4d7c131be4ac988b4d651b395c33d
I20260927 12:41:41.424696 9693 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:41:41.447002 9693 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:41:41.447942 9693 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:41:41.464634 9693 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1becf9c54e034bf2a4716a7209ed75a7: Generated new TSK 0
I20260927 12:41:41.465411 9693 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:41:41.476135 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512840836255-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512840836255-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512840836255-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38659
--builtin_ntp_servers=127.0.98.20:33775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260927 12:41:41.595352 9703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:41:41.595626 9703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:41:41.595691 9703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:41:41.599641 9703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:41:41.599690 9703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:41:41.599777 9703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:41:41.604544 9703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512840836255-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512840836255-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512840836255-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:41:41.605648 9703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:41:41.608327 9703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:41:41.622820 9711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:41.622970 9703 server_base.cc:1034] running on GCE node
W20260927 12:41:41.622809 9709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:41.622809 9708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:41.623624 9703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:41:41.624310 9703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:41:41.625479 9703 hybrid_clock.cc:648] HybridClock initialized: now 1790512901625459 us; error 38 us; skew 500 ppm
I20260927 12:41:41.628149 9703 webserver.cc:460] Webserver started at http://127.0.98.1:33389/ using document root <none> and password file <none>
I20260927 12:41:41.628906 9703 fs_manager.cc:359] Metadata directory not provided
I20260927 12:41:41.629017 9703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:41:41.629272 9703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:41:41.632037 9703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512840836255-392-0/minicluster-data/ts-0/data/instance:
uuid: "596dd69d146e4252a27257ce97dbec2e"
format_stamp: "Formatted at 2026-09-27 12:41:41 on dist-test-slave-bnc1"
server_key: "f65075dbbf40dafcfa5ddcf029e0d5e9"
server_key_iv: "4bfabcaf9639b9e94c0d9a0823934448"
server_key_version: "encryptionkey@0"
I20260927 12:41:41.632694 9703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512840836255-392-0/minicluster-data/ts-0/wal/instance:
uuid: "596dd69d146e4252a27257ce97dbec2e"
format_stamp: "Formatted at 2026-09-27 12:41:41 on dist-test-slave-bnc1"
server_key: "f65075dbbf40dafcfa5ddcf029e0d5e9"
server_key_iv: "4bfabcaf9639b9e94c0d9a0823934448"
server_key_version: "encryptionkey@0"
I20260927 12:41:41.636355 9703 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 12:41:41.638902 9717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:41.640105 9703 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 12:41:41.640233 9703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512840836255-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512840836255-392-0/minicluster-data/ts-0/wal
uuid: "596dd69d146e4252a27257ce97dbec2e"
format_stamp: "Formatted at 2026-09-27 12:41:41 on dist-test-slave-bnc1"
server_key: "f65075dbbf40dafcfa5ddcf029e0d5e9"
server_key_iv: "4bfabcaf9639b9e94c0d9a0823934448"
server_key_version: "encryptionkey@0"
I20260927 12:41:41.640342 9703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512840836255-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512840836255-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512840836255-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:41:41.670934 9703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:41:41.671720 9703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:41:41.671917 9703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:41:41.672518 9703 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:41:41.673511 9703 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:41:41.673597 9703 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:41.673673 9703 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:41:41.673708 9703 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:41.682047 9703 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34191
I20260927 12:41:41.682075 9830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34191 every 8 connection(s)
I20260927 12:41:41.682937 9703 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:41:41.683157 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 9703
I20260927 12:41:41.683269 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512840836255-392-0/minicluster-data/ts-0/wal/instance
I20260927 12:41:41.683606 392 external_mini_cluster.cc:1428] Setting key dc7a5ff1956af0d6d077f6da03caffc3
I20260927 12:41:41.685766 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512840836255-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512840836255-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512840836255-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38659
--builtin_ntp_servers=127.0.98.20:33775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 12:41:41.692173 9682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:41.684722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52543 (local address 127.0.98.62:38659)
0927 12:41:41.684870 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:41.684876 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:41.686253 (+ 1377us) server_negotiation.cc:407] Connection header received
0927 12:41:41.687607 (+ 1354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:41.687610 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:41.687661 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:41.687739 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:41.688586 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:41.689120 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:41.690276 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:41.690504 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:41.690613 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:41.691679 (+ 1066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:41.691683 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:41.691685 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:41:41.691698 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:41.691725 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:41:41.691733 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:41.691806 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:41.691971 (+ 165us) server_negotiation.cc:299] Negotiation successful
0927 12:41:41.692027 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 12:41:41.692926 9833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:41.685326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38659 (local address 127.0.98.1:52543)
0927 12:41:41.686126 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:41.686163 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:41.687404 (+ 1241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:41.687912 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:41.687918 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:41.687996 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:41.688442 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:41.688453 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:41.689250 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:41.689254 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:41.690161 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:41.690171 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:41.690654 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:41.690658 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:41.690809 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:41.691391 (+ 582us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 12:41:41.691393 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:41.691429 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 12:41:41.691487 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:41:41.691823 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:41:41.691831 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:41:41.691837 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 12:41:41.691958 (+ 121us) client_negotiation.cc:240] Negotiation successful
0927 12:41:41.692213 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":159,"threads_started":1}
I20260927 12:41:41.694054 9831 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38659
I20260927 12:41:41.694291 9831 heartbeater.cc:464] Registering TS with master...
I20260927 12:41:41.694758 9831 heartbeater.cc:511] Master 127.0.98.62:38659 requested a full tablet report, sending...
I20260927 12:41:41.696116 9643 ts_manager.cc:194] Registered new tserver with Master: 596dd69d146e4252a27257ce97dbec2e (127.0.98.1:34191)
I20260927 12:41:41.697203 9643 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:52543
W20260927 12:41:41.795302 9834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:41:41.795574 9834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:41:41.795637 9834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:41:41.799645 9834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:41:41.799731 9834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:41:41.799818 9834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:41:41.804612 9834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512840836255-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512840836255-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512840836255-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:41:41.805714 9834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:41:41.807116 9834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:41:41.817059 9839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:41.817059 9840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:41.817080 9842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:41.817405 9834 server_base.cc:1034] running on GCE node
I20260927 12:41:41.817838 9834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:41:41.818492 9834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:41:41.819684 9834 hybrid_clock.cc:648] HybridClock initialized: now 1790512901819669 us; error 31 us; skew 500 ppm
I20260927 12:41:41.821388 9834 webserver.cc:460] Webserver started at http://127.0.98.2:39037/ using document root <none> and password file <none>
I20260927 12:41:41.822029 9834 fs_manager.cc:359] Metadata directory not provided
I20260927 12:41:41.822124 9834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:41:41.822324 9834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:41:41.824393 9834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512840836255-392-0/minicluster-data/ts-1/data/instance:
uuid: "266f9473e5b347aaa18b36df31a4ff2d"
format_stamp: "Formatted at 2026-09-27 12:41:41 on dist-test-slave-bnc1"
server_key: "0bccd4e7d3478f9ecd5266183a459e46"
server_key_iv: "25bbd6c34de711fca386f37373a1406d"
server_key_version: "encryptionkey@0"
I20260927 12:41:41.824971 9834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512840836255-392-0/minicluster-data/ts-1/wal/instance:
uuid: "266f9473e5b347aaa18b36df31a4ff2d"
format_stamp: "Formatted at 2026-09-27 12:41:41 on dist-test-slave-bnc1"
server_key: "0bccd4e7d3478f9ecd5266183a459e46"
server_key_iv: "25bbd6c34de711fca386f37373a1406d"
server_key_version: "encryptionkey@0"
I20260927 12:41:41.828660 9834 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 12:41:41.831089 9848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:41.832325 9834 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 12:41:41.832459 9834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512840836255-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512840836255-392-0/minicluster-data/ts-1/wal
uuid: "266f9473e5b347aaa18b36df31a4ff2d"
format_stamp: "Formatted at 2026-09-27 12:41:41 on dist-test-slave-bnc1"
server_key: "0bccd4e7d3478f9ecd5266183a459e46"
server_key_iv: "25bbd6c34de711fca386f37373a1406d"
server_key_version: "encryptionkey@0"
I20260927 12:41:41.832583 9834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512840836255-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512840836255-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512840836255-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:41:41.861094 9834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:41:41.861980 9834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:41:41.862195 9834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:41:41.862799 9834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:41:41.863756 9834 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:41:41.863829 9834 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:41.863900 9834 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:41:41.863951 9834 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:41.872434 9834 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33883
I20260927 12:41:41.872462 9961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33883 every 8 connection(s)
I20260927 12:41:41.873306 9834 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:41:41.882161 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 9834
I20260927 12:41:41.882261 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512840836255-392-0/minicluster-data/ts-1/wal/instance
I20260927 12:41:41.882619 392 external_mini_cluster.cc:1428] Setting key 21e6fecdf96da5b4e7784c32106fb46c
I20260927 12:41:41.882769 9682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:41.875063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46427 (local address 127.0.98.62:38659)
0927 12:41:41.875196 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:41.875200 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:41.876230 (+ 1030us) server_negotiation.cc:407] Connection header received
0927 12:41:41.877660 (+ 1430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:41.877663 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:41.877714 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:41.877801 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:41.878672 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:41.879195 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:41.880302 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:41.880527 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:41.880639 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:41.881817 (+ 1178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:41.881820 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:41.881823 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:41:41.881834 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:41.881853 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:41:41.881861 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:41.881941 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:41.882548 (+ 607us) server_negotiation.cc:299] Negotiation successful
0927 12:41:41.882616 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 12:41:41.883811 9964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:41.875477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38659 (local address 127.0.98.2:46427)
0927 12:41:41.876088 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:41.876130 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:41.877427 (+ 1297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:41.877976 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:41.877982 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:41.878063 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:41.878521 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:41.878533 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:41.879317 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:41.879320 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:41.880182 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:41.880190 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:41.880681 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:41.880685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:41.880851 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:41.881483 (+ 632us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 12:41:41.881485 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:41.881524 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 12:41:41.881650 (+ 126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:41:41.882359 (+ 709us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:41:41.882372 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:41:41.882380 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 12:41:41.882512 (+ 132us) client_negotiation.cc:240] Negotiation successful
0927 12:41:41.882800 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":105,"threads_started":1}
I20260927 12:41:41.884657 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512840836255-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512840836255-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512840836255-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38659
--builtin_ntp_servers=127.0.98.20:33775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 12:41:41.885139 9962 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38659
I20260927 12:41:41.885386 9962 heartbeater.cc:464] Registering TS with master...
I20260927 12:41:41.886013 9962 heartbeater.cc:511] Master 127.0.98.62:38659 requested a full tablet report, sending...
I20260927 12:41:41.887043 9643 ts_manager.cc:194] Registered new tserver with Master: 266f9473e5b347aaa18b36df31a4ff2d (127.0.98.2:33883)
I20260927 12:41:41.887601 9643 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:46427
W20260927 12:41:41.996541 9965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:41:41.996848 9965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:41:41.996923 9965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:41:42.002092 9965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:41:42.002162 9965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:41:42.002250 9965 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:41:42.007081 9965 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512840836255-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512840836255-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512840836255-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:41:42.008140 9965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:41:42.009526 9965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:41:42.019121 9970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:42.019129 9971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:42.019143 9973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:42.019346 9965 server_base.cc:1034] running on GCE node
I20260927 12:41:42.019776 9965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:41:42.020437 9965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:41:42.021649 9965 hybrid_clock.cc:648] HybridClock initialized: now 1790512902021631 us; error 30 us; skew 500 ppm
I20260927 12:41:42.023442 9965 webserver.cc:460] Webserver started at http://127.0.98.3:44187/ using document root <none> and password file <none>
I20260927 12:41:42.024027 9965 fs_manager.cc:359] Metadata directory not provided
I20260927 12:41:42.024117 9965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:41:42.024402 9965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:41:42.026503 9965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512840836255-392-0/minicluster-data/ts-2/data/instance:
uuid: "ac45a1b760254faa86c25c19349e715f"
format_stamp: "Formatted at 2026-09-27 12:41:42 on dist-test-slave-bnc1"
server_key: "93d72894aefee31286c47d9c2147f9bf"
server_key_iv: "38783948e22dc6d5a1fa620332fe9e05"
server_key_version: "encryptionkey@0"
I20260927 12:41:42.027067 9965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512840836255-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ac45a1b760254faa86c25c19349e715f"
format_stamp: "Formatted at 2026-09-27 12:41:42 on dist-test-slave-bnc1"
server_key: "93d72894aefee31286c47d9c2147f9bf"
server_key_iv: "38783948e22dc6d5a1fa620332fe9e05"
server_key_version: "encryptionkey@0"
I20260927 12:41:42.030594 9965 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20260927 12:41:42.032920 9979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:42.034103 9965 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 12:41:42.034227 9965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512840836255-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512840836255-392-0/minicluster-data/ts-2/wal
uuid: "ac45a1b760254faa86c25c19349e715f"
format_stamp: "Formatted at 2026-09-27 12:41:42 on dist-test-slave-bnc1"
server_key: "93d72894aefee31286c47d9c2147f9bf"
server_key_iv: "38783948e22dc6d5a1fa620332fe9e05"
server_key_version: "encryptionkey@0"
I20260927 12:41:42.034330 9965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512840836255-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512840836255-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512840836255-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:41:42.051826 9965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:41:42.052420 9965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:41:42.052603 9965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:41:42.053129 9965 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:41:42.054051 9965 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:41:42.054126 9965 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:42.054191 9965 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:41:42.054244 9965 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:42.063416 9965 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36653
I20260927 12:41:42.063434 10092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36653 every 8 connection(s)
I20260927 12:41:42.064337 9965 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:41:42.071516 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 9965
I20260927 12:41:42.071609 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790512840836255-392-0/minicluster-data/ts-2/wal/instance
I20260927 12:41:42.071909 392 external_mini_cluster.cc:1428] Setting key b9fd02be84d4c938acee57b60b6dd395
I20260927 12:41:42.073628 9682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:42.066192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49911 (local address 127.0.98.62:38659)
0927 12:41:42.066356 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:42.066360 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:42.067371 (+ 1011us) server_negotiation.cc:407] Connection header received
0927 12:41:42.068736 (+ 1365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:42.068740 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:42.068795 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:42.068906 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:42.069802 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:42.070329 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:42.071529 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:42.071820 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:42.071905 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:42.073118 (+ 1213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:42.073123 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:42.073126 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:41:42.073139 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:42.073160 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:41:42.073169 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:42.073262 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:42.073417 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 12:41:42.073471 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 12:41:42.074519 10095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:42.066622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38659 (local address 127.0.98.3:49911)
0927 12:41:42.067229 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:42.067266 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:42.068527 (+ 1261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:42.069073 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:42.069080 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:42.069161 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:42.069662 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:42.069675 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:42.070451 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:42.070455 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:42.071384 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:42.071396 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:42.071966 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:42.071971 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:42.072166 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:42.072807 (+ 641us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 12:41:42.072809 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:42.072861 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 12:41:42.072935 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:41:42.073277 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:41:42.073292 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:41:42.073296 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 12:41:42.073394 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 12:41:42.073659 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":103,"threads_started":1}
I20260927 12:41:42.075536 10093 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38659
I20260927 12:41:42.075718 10093 heartbeater.cc:464] Registering TS with master...
I20260927 12:41:42.076134 10093 heartbeater.cc:511] Master 127.0.98.62:38659 requested a full tablet report, sending...
I20260927 12:41:42.077009 9643 ts_manager.cc:194] Registered new tserver with Master: ac45a1b760254faa86c25c19349e715f (127.0.98.3:36653)
I20260927 12:41:42.077608 9643 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:49911
I20260927 12:41:42.085696 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 12:41:42.088292 9682 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:41:42.087577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43308 (local address 127.0.98.62:38659)
0927 12:41:42.087706 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:42.087709 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:42.087799 (+ 90us) server_negotiation.cc:407] Connection header received
0927 12:41:42.087880 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:42.087882 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:42.087920 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:42.087986 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:42.088198 (+ 212us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43308: BlockingRecv error: recv got EOF from 127.0.0.1:43308 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 12:41:42.088498 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 9703
I20260927 12:41:42.093950 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 9834
I20260927 12:41:42.098856 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 9965
I20260927 12:41:42.104197 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 9612
2026-09-27T12:41:42Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (917 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-27T12:41:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:41:42Z Disabled control of system clock
I20260927 12:41:42.133419 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512840836255-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512840836255-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512840836255-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38521
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38521
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 12:41:42.240075 10104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:41:42.240337 10104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:41:42.240373 10104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:41:42.244342 10104 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:41:42.244392 10104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:41:42.244410 10104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:41:42.244426 10104 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:41:42.244481 10104 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:41:42.249034 10104 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512840836255-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512840836255-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38521
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512840836255-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:41:42.250128 10104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:41:42.251477 10104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 12:41:42.259692 10104 server_base.cc:1034] running on GCE node
W20260927 12:41:42.259666 10112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:42.259625 10110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:42.259641 10109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:42.260358 10104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:41:42.261444 10104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:41:42.262646 10104 hybrid_clock.cc:648] HybridClock initialized: now 1790512902262629 us; error 27 us; skew 500 ppm
I20260927 12:41:42.264437 10104 webserver.cc:460] Webserver started at http://127.0.98.62:32993/ using document root <none> and password file <none>
I20260927 12:41:42.265041 10104 fs_manager.cc:359] Metadata directory not provided
I20260927 12:41:42.265185 10104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:41:42.265413 10104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:41:42.267473 10104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512840836255-392-0/minicluster-data/master-0/data/instance:
uuid: "36a67697cf074a0eb3d08f6df77bc888"
format_stamp: "Formatted at 2026-09-27 12:41:42 on dist-test-slave-bnc1"
server_key: "555401208ad4dc659d9ffa85b437343b"
server_key_iv: "e686162c29ba351027d51fac2ff8c70d"
server_key_version: "encryptionkey@0"
I20260927 12:41:42.268041 10104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512840836255-392-0/minicluster-data/master-0/wal/instance:
uuid: "36a67697cf074a0eb3d08f6df77bc888"
format_stamp: "Formatted at 2026-09-27 12:41:42 on dist-test-slave-bnc1"
server_key: "555401208ad4dc659d9ffa85b437343b"
server_key_iv: "e686162c29ba351027d51fac2ff8c70d"
server_key_version: "encryptionkey@0"
I20260927 12:41:42.271605 10104 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 12:41:42.273888 10118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:42.275079 10104 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 12:41:42.275234 10104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512840836255-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512840836255-392-0/minicluster-data/master-0/wal
uuid: "36a67697cf074a0eb3d08f6df77bc888"
format_stamp: "Formatted at 2026-09-27 12:41:42 on dist-test-slave-bnc1"
server_key: "555401208ad4dc659d9ffa85b437343b"
server_key_iv: "e686162c29ba351027d51fac2ff8c70d"
server_key_version: "encryptionkey@0"
I20260927 12:41:42.275342 10104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512840836255-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512840836255-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512840836255-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:41:42.279762 10104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:41:42.280329 10104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:41:42.280504 10104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:41:42.286984 10104 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38521
I20260927 12:41:42.287005 10170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38521 every 8 connection(s)
I20260927 12:41:42.287927 10104 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:41:42.288305 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 10104
I20260927 12:41:42.288441 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512840836255-392-0/minicluster-data/master-0/wal/instance
I20260927 12:41:42.288780 392 external_mini_cluster.cc:1428] Setting key 7f7e2b0aa0fef64fb7b5d0af9e1d1e11
I20260927 12:41:42.291440 10171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:41:42.295142 10174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:42.290255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43650 (local address 127.0.98.62:38521)
0927 12:41:42.291523 (+ 1268us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:42.291547 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:42.291602 (+ 55us) server_negotiation.cc:407] Connection header received
0927 12:41:42.292293 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:42.292309 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:42.292660 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:42.293086 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:42.293883 (+ 797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:42.293888 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:42.293899 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:41:42.293909 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:42.293988 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:41:42.294007 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:42.294174 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:42.294250 (+ 76us) server_negotiation.cc:299] Negotiation successful
0927 12:41:42.294478 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":876,"thread_start_us":694,"threads_started":1}
I20260927 12:41:42.297415 10171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36a67697cf074a0eb3d08f6df77bc888: Bootstrap starting.
I20260927 12:41:42.299672 10171 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36a67697cf074a0eb3d08f6df77bc888: Neither blocks nor log segments found. Creating new log.
I20260927 12:41:42.300371 10171 log.cc:824] T 00000000000000000000000000000000 P 36a67697cf074a0eb3d08f6df77bc888: Log is configured to *not* fsync() on all Append() calls
I20260927 12:41:42.302263 10171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36a67697cf074a0eb3d08f6df77bc888: No bootstrap required, opened a new log
I20260927 12:41:42.305699 10171 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36a67697cf074a0eb3d08f6df77bc888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36a67697cf074a0eb3d08f6df77bc888" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38521 } }
I20260927 12:41:42.305968 10171 raft_consensus.cc:374] T 00000000000000000000000000000000 P 36a67697cf074a0eb3d08f6df77bc888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:41:42.306033 10171 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36a67697cf074a0eb3d08f6df77bc888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36a67697cf074a0eb3d08f6df77bc888, State: Initialized, Role: FOLLOWER
I20260927 12:41:42.306468 10171 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36a67697cf074a0eb3d08f6df77bc888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36a67697cf074a0eb3d08f6df77bc888" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38521 } }
I20260927 12:41:42.306604 10171 raft_consensus.cc:388] T 00000000000000000000000000000000 P 36a67697cf074a0eb3d08f6df77bc888 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:41:42.306665 10171 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36a67697cf074a0eb3d08f6df77bc888 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:41:42.306764 10171 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36a67697cf074a0eb3d08f6df77bc888 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:41:42.307605 10171 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36a67697cf074a0eb3d08f6df77bc888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36a67697cf074a0eb3d08f6df77bc888" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38521 } }
I20260927 12:41:42.307930 10171 leader_election.cc:302] T 00000000000000000000000000000000 P 36a67697cf074a0eb3d08f6df77bc888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36a67697cf074a0eb3d08f6df77bc888; no voters:
I20260927 12:41:42.308226 10171 leader_election.cc:288] T 00000000000000000000000000000000 P 36a67697cf074a0eb3d08f6df77bc888 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:41:42.308352 10176 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36a67697cf074a0eb3d08f6df77bc888 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:41:42.308616 10176 raft_consensus.cc:686] T 00000000000000000000000000000000 P 36a67697cf074a0eb3d08f6df77bc888 [term 1 LEADER]: Becoming Leader. State: Replica: 36a67697cf074a0eb3d08f6df77bc888, State: Running, Role: LEADER
I20260927 12:41:42.308926 10176 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36a67697cf074a0eb3d08f6df77bc888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36a67697cf074a0eb3d08f6df77bc888" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38521 } }
I20260927 12:41:42.309404 10171 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36a67697cf074a0eb3d08f6df77bc888 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:41:42.310748 10177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36a67697cf074a0eb3d08f6df77bc888 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "36a67697cf074a0eb3d08f6df77bc888" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36a67697cf074a0eb3d08f6df77bc888" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38521 } } }
I20260927 12:41:42.310763 10178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36a67697cf074a0eb3d08f6df77bc888 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36a67697cf074a0eb3d08f6df77bc888. Latest consensus state: current_term: 1 leader_uuid: "36a67697cf074a0eb3d08f6df77bc888" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36a67697cf074a0eb3d08f6df77bc888" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38521 } } }
I20260927 12:41:42.310877 10177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36a67697cf074a0eb3d08f6df77bc888 [sys.catalog]: This master's current role is: LEADER
I20260927 12:41:42.310880 10178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36a67697cf074a0eb3d08f6df77bc888 [sys.catalog]: This master's current role is: LEADER
I20260927 12:41:42.311236 10185 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:41:42.314072 10185 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:41:42.318907 10185 catalog_manager.cc:1347] Generated new cluster ID: bda801e80d4344cf91c3312eb31fa1e7
I20260927 12:41:42.318964 10185 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:41:42.331996 10185 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:41:42.332327 10185 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:41:42.340386 10185 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 36a67697cf074a0eb3d08f6df77bc888: Generated new TSK 0
I20260927 12:41:42.341187 10185 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:41:42.353374 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512840836255-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512840836255-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512840836255-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38521
--builtin_ntp_servers=127.0.98.20:45787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 12:41:42.462200 10195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:41:42.462455 10195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:41:42.462495 10195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:41:42.466598 10195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:41:42.466668 10195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:41:42.466766 10195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:41:42.471832 10195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512840836255-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512840836255-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512840836255-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:41:42.472954 10195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:41:42.474381 10195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:41:42.483973 10201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:42.484021 10200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:42.484002 10203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:42.484140 10195 server_base.cc:1034] running on GCE node
I20260927 12:41:42.484679 10195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:41:42.485395 10195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:41:42.486578 10195 hybrid_clock.cc:648] HybridClock initialized: now 1790512902486544 us; error 64 us; skew 500 ppm
I20260927 12:41:42.488507 10195 webserver.cc:460] Webserver started at http://127.0.98.1:46379/ using document root <none> and password file <none>
I20260927 12:41:42.489133 10195 fs_manager.cc:359] Metadata directory not provided
I20260927 12:41:42.489223 10195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:41:42.489455 10195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:41:42.491699 10195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512840836255-392-0/minicluster-data/ts-0/data/instance:
uuid: "8e00a07b27bb40828fb7009ae33a3fda"
format_stamp: "Formatted at 2026-09-27 12:41:42 on dist-test-slave-bnc1"
server_key: "ef046ff8157c4e59d77766bf7b403f86"
server_key_iv: "9747022655cc6d677076d5c0058ca6bb"
server_key_version: "encryptionkey@0"
I20260927 12:41:42.492321 10195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512840836255-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8e00a07b27bb40828fb7009ae33a3fda"
format_stamp: "Formatted at 2026-09-27 12:41:42 on dist-test-slave-bnc1"
server_key: "ef046ff8157c4e59d77766bf7b403f86"
server_key_iv: "9747022655cc6d677076d5c0058ca6bb"
server_key_version: "encryptionkey@0"
I20260927 12:41:42.495954 10195 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 12:41:42.498301 10209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:42.499491 10195 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 12:41:42.499621 10195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512840836255-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512840836255-392-0/minicluster-data/ts-0/wal
uuid: "8e00a07b27bb40828fb7009ae33a3fda"
format_stamp: "Formatted at 2026-09-27 12:41:42 on dist-test-slave-bnc1"
server_key: "ef046ff8157c4e59d77766bf7b403f86"
server_key_iv: "9747022655cc6d677076d5c0058ca6bb"
server_key_version: "encryptionkey@0"
I20260927 12:41:42.499742 10195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512840836255-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512840836255-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512840836255-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:41:42.504266 10195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:41:42.504843 10195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:41:42.505021 10195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:41:42.505551 10195 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:41:42.506464 10195 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:41:42.506532 10195 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:42.506593 10195 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:41:42.506644 10195 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:42.515537 10195 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41709
I20260927 12:41:42.515590 10322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41709 every 8 connection(s)
I20260927 12:41:42.516422 10195 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:41:42.519663 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 10195
I20260927 12:41:42.519753 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512840836255-392-0/minicluster-data/ts-0/wal/instance
I20260927 12:41:42.520027 392 external_mini_cluster.cc:1428] Setting key c52e45d23f566473fd5d4c95516a15ac
I20260927 12:41:42.521871 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512840836255-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512840836255-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512840836255-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38521
--builtin_ntp_servers=127.0.98.20:45787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 12:41:42.522580 10174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:42.518174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47547 (local address 127.0.98.62:38521)
0927 12:41:42.518303 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:42.518307 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:42.519337 (+ 1030us) server_negotiation.cc:407] Connection header received
0927 12:41:42.520727 (+ 1390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:42.520730 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:42.520773 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:42.520851 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:42.522111 (+ 1260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:42.522114 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:42.522116 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:41:42.522128 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:42.522147 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:41:42.522155 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:42.522249 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:42.522403 (+ 154us) server_negotiation.cc:299] Negotiation successful
0927 12:41:42.522456 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 12:41:42.523402 10325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:42.518563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38521 (local address 127.0.98.1:47547)
0927 12:41:42.519189 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:42.519228 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:42.520524 (+ 1296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:42.521004 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:42.521010 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:42.521095 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:42.521741 (+ 646us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 12:41:42.521743 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:42.521784 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 12:41:42.521850 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:41:42.522248 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:41:42.522258 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:41:42.522266 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 12:41:42.522455 (+ 189us) client_negotiation.cc:240] Negotiation successful
0927 12:41:42.522698 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":119,"threads_started":1}
I20260927 12:41:42.524608 10323 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38521
I20260927 12:41:42.524842 10323 heartbeater.cc:464] Registering TS with master...
I20260927 12:41:42.525290 10323 heartbeater.cc:511] Master 127.0.98.62:38521 requested a full tablet report, sending...
I20260927 12:41:42.526834 10135 ts_manager.cc:194] Registered new tserver with Master: 8e00a07b27bb40828fb7009ae33a3fda (127.0.98.1:41709)
W20260927 12:41:42.635694 10326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:41:42.635952 10326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:41:42.636013 10326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:41:42.640045 10326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:41:42.640105 10326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:41:42.640201 10326 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:41:42.644963 10326 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512840836255-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512840836255-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10326
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512840836255-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:41:42.646109 10326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:41:42.647466 10326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:41:42.656916 10332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:42.657022 10326 server_base.cc:1034] running on GCE node
W20260927 12:41:42.656962 10331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:42.656970 10334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:42.657639 10326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:41:42.658263 10326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:41:42.659437 10326 hybrid_clock.cc:648] HybridClock initialized: now 1790512902659403 us; error 48 us; skew 500 ppm
I20260927 12:41:42.661211 10326 webserver.cc:460] Webserver started at http://127.0.98.2:38391/ using document root <none> and password file <none>
I20260927 12:41:42.661868 10326 fs_manager.cc:359] Metadata directory not provided
I20260927 12:41:42.661957 10326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:41:42.662184 10326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:41:42.664212 10326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512840836255-392-0/minicluster-data/ts-1/data/instance:
uuid: "b385a42b53704397b3417fd8cfed5278"
format_stamp: "Formatted at 2026-09-27 12:41:42 on dist-test-slave-bnc1"
server_key: "f974b8c9811083b45261d6d27335b6ea"
server_key_iv: "33d9b19cd6ba98596a82a549b8ebfe7c"
server_key_version: "encryptionkey@0"
I20260927 12:41:42.664772 10326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512840836255-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b385a42b53704397b3417fd8cfed5278"
format_stamp: "Formatted at 2026-09-27 12:41:42 on dist-test-slave-bnc1"
server_key: "f974b8c9811083b45261d6d27335b6ea"
server_key_iv: "33d9b19cd6ba98596a82a549b8ebfe7c"
server_key_version: "encryptionkey@0"
I20260927 12:41:42.668522 10326 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 12:41:42.670845 10340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:42.671964 10326 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 12:41:42.672089 10326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512840836255-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512840836255-392-0/minicluster-data/ts-1/wal
uuid: "b385a42b53704397b3417fd8cfed5278"
format_stamp: "Formatted at 2026-09-27 12:41:42 on dist-test-slave-bnc1"
server_key: "f974b8c9811083b45261d6d27335b6ea"
server_key_iv: "33d9b19cd6ba98596a82a549b8ebfe7c"
server_key_version: "encryptionkey@0"
I20260927 12:41:42.672192 10326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512840836255-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512840836255-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512840836255-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:41:42.676558 10326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:41:42.677102 10326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:41:42.677278 10326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:41:42.677896 10326 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:41:42.678848 10326 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:41:42.678920 10326 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:42.678987 10326 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:41:42.679019 10326 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:42.687330 10326 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43073
I20260927 12:41:42.687359 10453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43073 every 8 connection(s)
I20260927 12:41:42.688203 10326 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:41:42.693928 10174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:42.689935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50683 (local address 127.0.98.62:38521)
0927 12:41:42.690092 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:42.690097 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:42.691142 (+ 1045us) server_negotiation.cc:407] Connection header received
0927 12:41:42.692402 (+ 1260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:42.692405 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:42.692441 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:42.692515 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:42.693517 (+ 1002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:42.693520 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:42.693522 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:41:42.693533 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:42.693552 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:41:42.693560 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:42.693684 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:42.693790 (+ 106us) server_negotiation.cc:299] Negotiation successful
0927 12:41:42.693827 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 12:41:42.694747 10456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:42.690356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38521 (local address 127.0.98.2:50683)
0927 12:41:42.690998 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:42.691038 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:42.692219 (+ 1181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:42.692668 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:42.692674 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:42.692761 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:42.693280 (+ 519us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 12:41:42.693282 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:42.693320 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 12:41:42.693385 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:41:42.693670 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:41:42.693676 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:41:42.693680 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 12:41:42.693814 (+ 134us) client_negotiation.cc:240] Negotiation successful
0927 12:41:42.693992 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":119,"threads_started":1}
I20260927 12:41:42.695724 10454 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38521
I20260927 12:41:42.695957 10454 heartbeater.cc:464] Registering TS with master...
I20260927 12:41:42.696358 10454 heartbeater.cc:511] Master 127.0.98.62:38521 requested a full tablet report, sending...
I20260927 12:41:42.697275 10135 ts_manager.cc:194] Registered new tserver with Master: b385a42b53704397b3417fd8cfed5278 (127.0.98.2:43073)
I20260927 12:41:42.697918 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 10326
I20260927 12:41:42.698053 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512840836255-392-0/minicluster-data/ts-1/wal/instance
I20260927 12:41:42.698341 392 external_mini_cluster.cc:1428] Setting key d35e92e3ab3aa99e784bfcf8591f9cc0
I20260927 12:41:42.700037 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512840836255-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512840836255-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512840836255-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38521
--builtin_ntp_servers=127.0.98.20:45787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 12:41:42.811117 10457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:41:42.811386 10457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:41:42.811426 10457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:41:42.815786 10457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:41:42.815845 10457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:41:42.815902 10457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:41:42.821264 10457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512840836255-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512840836255-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10457
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512840836255-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:41:42.822434 10457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:41:42.823807 10457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:41:42.833734 10462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:42.833734 10465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:42.833760 10463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:42.833940 10457 server_base.cc:1034] running on GCE node
I20260927 12:41:42.834447 10457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:41:42.835129 10457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:41:42.836329 10457 hybrid_clock.cc:648] HybridClock initialized: now 1790512902836313 us; error 28 us; skew 500 ppm
I20260927 12:41:42.838157 10457 webserver.cc:460] Webserver started at http://127.0.98.3:40165/ using document root <none> and password file <none>
I20260927 12:41:42.838737 10457 fs_manager.cc:359] Metadata directory not provided
I20260927 12:41:42.838800 10457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:41:42.839051 10457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:41:42.841216 10457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512840836255-392-0/minicluster-data/ts-2/data/instance:
uuid: "4a3073b25ae34868a17688cc40f5add7"
format_stamp: "Formatted at 2026-09-27 12:41:42 on dist-test-slave-bnc1"
server_key: "c93172094647d2c2145f619ade2bacf5"
server_key_iv: "a67550b63b4fdaaf8d654e62b4dd816c"
server_key_version: "encryptionkey@0"
I20260927 12:41:42.841814 10457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512840836255-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4a3073b25ae34868a17688cc40f5add7"
format_stamp: "Formatted at 2026-09-27 12:41:42 on dist-test-slave-bnc1"
server_key: "c93172094647d2c2145f619ade2bacf5"
server_key_iv: "a67550b63b4fdaaf8d654e62b4dd816c"
server_key_version: "encryptionkey@0"
I20260927 12:41:42.845474 10457 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:41:42.847849 10471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:42.849066 10457 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:41:42.849159 10457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512840836255-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512840836255-392-0/minicluster-data/ts-2/wal
uuid: "4a3073b25ae34868a17688cc40f5add7"
format_stamp: "Formatted at 2026-09-27 12:41:42 on dist-test-slave-bnc1"
server_key: "c93172094647d2c2145f619ade2bacf5"
server_key_iv: "a67550b63b4fdaaf8d654e62b4dd816c"
server_key_version: "encryptionkey@0"
I20260927 12:41:42.849272 10457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512840836255-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512840836255-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512840836255-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:41:42.853791 10457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:41:42.854364 10457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:41:42.854498 10457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:41:42.854992 10457 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:41:42.855968 10457 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:41:42.856015 10457 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:42.856052 10457 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:41:42.856074 10457 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:42.864897 10457 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39345
I20260927 12:41:42.864917 10584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39345 every 8 connection(s)
I20260927 12:41:42.865970 10457 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:41:42.871917 10174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:42.867778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:32829 (local address 127.0.98.62:38521)
0927 12:41:42.867928 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:42.867931 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:42.869008 (+ 1077us) server_negotiation.cc:407] Connection header received
0927 12:41:42.870303 (+ 1295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:42.870306 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:42.870344 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:42.870420 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:42.871452 (+ 1032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:42.871455 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:42.871457 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:41:42.871467 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:42.871487 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:41:42.871495 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:42.871572 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:42.871751 (+ 179us) server_negotiation.cc:299] Negotiation successful
0927 12:41:42.871795 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 12:41:42.872709 10587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:42.868196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38521 (local address 127.0.98.3:32829)
0927 12:41:42.868855 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:42.868895 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:42.870117 (+ 1222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:42.870576 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:42.870583 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:42.870672 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:42.871206 (+ 534us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 12:41:42.871207 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:42.871246 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 12:41:42.871316 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:41:42.871571 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:41:42.871577 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:41:42.871580 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 12:41:42.871709 (+ 129us) client_negotiation.cc:240] Negotiation successful
0927 12:41:42.871912 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":114,"threads_started":1}
I20260927 12:41:42.873878 10585 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38521
I20260927 12:41:42.874068 10585 heartbeater.cc:464] Registering TS with master...
I20260927 12:41:42.874418 10585 heartbeater.cc:511] Master 127.0.98.62:38521 requested a full tablet report, sending...
I20260927 12:41:42.875345 10135 ts_manager.cc:194] Registered new tserver with Master: 4a3073b25ae34868a17688cc40f5add7 (127.0.98.3:39345)
I20260927 12:41:42.875989 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 10457
I20260927 12:41:42.876101 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790512840836255-392-0/minicluster-data/ts-2/wal/instance
I20260927 12:41:42.876470 392 external_mini_cluster.cc:1428] Setting key e31b58236c6df8e83e754bb0f40186df
I20260927 12:41:42.878275 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 12:41:42.881137 10174 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:41:42.880168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43666 (local address 127.0.98.62:38521)
0927 12:41:42.880313 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:42.880318 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:42.880506 (+ 188us) server_negotiation.cc:407] Connection header received
0927 12:41:42.880652 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:42.880656 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:42.880699 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:42.880784 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:42.881014 (+ 230us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43666: BlockingRecv error: recv got EOF from 127.0.0.1:43666 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 12:41:42.881494 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 10195
I20260927 12:41:42.888221 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 10326
I20260927 12:41:42.893254 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 10457
I20260927 12:41:42.898228 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 10104
2026-09-27T12:41:42Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (793 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:41:42 dist-test-slave-bnc1 krb5kdc[10595](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:41:42 dist-test-slave-bnc1 krb5kdc[10595](info): set up 2 sockets
Sep 27 12:41:42 dist-test-slave-bnc1 krb5kdc[10595](info): commencing operation
krb5kdc: starting...
W20260927 12:41:43.926755 392 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/krb5kdc/test-user.keytab.
Sep 27 12:41:44 dist-test-slave-bnc1 krb5kdc[10595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:41:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:41:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:41:44.048913 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43447
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43447
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:41:44.156333 10608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:41:44.156602 10608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:41:44.156642 10608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:41:44.160871 10608 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:41:44.160961 10608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:41:44.160992 10608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:41:44.161027 10608 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:41:44.161062 10608 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:41:44.166680 10608 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43447
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10608
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:41:44.167678 10608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:41:44.168633 10608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:41:44.177215 10616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:44.177215 10613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:44.177260 10614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:44.177439 10608 server_base.cc:1034] running on GCE node
I20260927 12:41:44.178102 10608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:41:44.179216 10608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:41:44.180418 10608 hybrid_clock.cc:648] HybridClock initialized: now 1790512904180415 us; error 42 us; skew 500 ppm
Sep 27 12:41:44 dist-test-slave-bnc1 krb5kdc[10595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512904, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:41:44.193892 10608 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:41:44.194921 10608 webserver.cc:460] Webserver started at http://127.0.98.62:42107/ using document root <none> and password file <none>
I20260927 12:41:44.195560 10608 fs_manager.cc:359] Metadata directory not provided
I20260927 12:41:44.195657 10608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:41:44.195899 10608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:41:44.198112 10608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/minicluster-data/master-0/data/instance:
uuid: "6971716913f14286bc4e04c193213b1b"
format_stamp: "Formatted at 2026-09-27 12:41:44 on dist-test-slave-bnc1"
server_key: "a6dc5e4cb0b1ece2f27c2e602f71be54"
server_key_iv: "33681525b81d6b483d11265da670cbad"
server_key_version: "encryptionkey@0"
I20260927 12:41:44.198699 10608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/minicluster-data/master-0/wal/instance:
uuid: "6971716913f14286bc4e04c193213b1b"
format_stamp: "Formatted at 2026-09-27 12:41:44 on dist-test-slave-bnc1"
server_key: "a6dc5e4cb0b1ece2f27c2e602f71be54"
server_key_iv: "33681525b81d6b483d11265da670cbad"
server_key_version: "encryptionkey@0"
I20260927 12:41:44.202672 10608 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 12:41:44.205031 10623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:44.206274 10608 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:41:44.206444 10608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/minicluster-data/master-0/wal
uuid: "6971716913f14286bc4e04c193213b1b"
format_stamp: "Formatted at 2026-09-27 12:41:44 on dist-test-slave-bnc1"
server_key: "a6dc5e4cb0b1ece2f27c2e602f71be54"
server_key_iv: "33681525b81d6b483d11265da670cbad"
server_key_version: "encryptionkey@0"
I20260927 12:41:44.206559 10608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:41:44.222718 10608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:41:44.225977 10608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:41:44.226182 10608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:41:44.233235 10608 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43447
I20260927 12:41:44.233496 10675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43447 every 8 connection(s)
I20260927 12:41:44.234222 10608 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:41:44.234772 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 10608
I20260927 12:41:44.234894 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/minicluster-data/master-0/wal/instance
I20260927 12:41:44.235178 392 external_mini_cluster.cc:1428] Setting key 8cf674669a9bc6c8d856044a055b947e
I20260927 12:41:44.237103 10676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:41:44.242532 10676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6971716913f14286bc4e04c193213b1b: Bootstrap starting.
Sep 27 12:41:44 dist-test-slave-bnc1 krb5kdc[10595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:41:44.244829 10676 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6971716913f14286bc4e04c193213b1b: Neither blocks nor log segments found. Creating new log.
I20260927 12:41:44.245653 10676 log.cc:824] T 00000000000000000000000000000000 P 6971716913f14286bc4e04c193213b1b: Log is configured to *not* fsync() on all Append() calls
I20260927 12:41:44.247484 10676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6971716913f14286bc4e04c193213b1b: No bootstrap required, opened a new log
I20260927 12:41:44.248656 10679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:44.236574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33074 (local address 127.0.98.62:43447)
0927 12:41:44.236955 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:44.236971 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:44.237005 (+ 34us) server_negotiation.cc:407] Connection header received
0927 12:41:44.237728 (+ 723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:44.237745 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:44.238060 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:44.238396 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:44.239156 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:44.240138 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:44.240976 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:44.241425 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:44.241610 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:44.244078 (+ 2468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:44.244083 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:44.244098 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:41:44.244123 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:44.245922 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:44.246361 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:44.246371 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:44.246375 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:44.246510 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:44.246787 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:44.246791 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:44.246792 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:44.247183 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:44.247390 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:44.247678 (+ 288us) server_negotiation.cc:299] Negotiation successful
0927 12:41:44.247962 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":125,"threads_started":1}
I20260927 12:41:44.251556 10676 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6971716913f14286bc4e04c193213b1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6971716913f14286bc4e04c193213b1b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43447 } }
I20260927 12:41:44.251792 10676 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6971716913f14286bc4e04c193213b1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:41:44.251878 10676 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6971716913f14286bc4e04c193213b1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6971716913f14286bc4e04c193213b1b, State: Initialized, Role: FOLLOWER
I20260927 12:41:44.252295 10676 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6971716913f14286bc4e04c193213b1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6971716913f14286bc4e04c193213b1b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43447 } }
I20260927 12:41:44.252444 10676 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6971716913f14286bc4e04c193213b1b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:41:44.252527 10676 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6971716913f14286bc4e04c193213b1b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:41:44.252643 10676 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6971716913f14286bc4e04c193213b1b [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:41:44.253714 10676 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6971716913f14286bc4e04c193213b1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6971716913f14286bc4e04c193213b1b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43447 } }
I20260927 12:41:44.254046 10676 leader_election.cc:302] T 00000000000000000000000000000000 P 6971716913f14286bc4e04c193213b1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6971716913f14286bc4e04c193213b1b; no voters:
I20260927 12:41:44.254387 10676 leader_election.cc:288] T 00000000000000000000000000000000 P 6971716913f14286bc4e04c193213b1b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:41:44.254518 10681 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6971716913f14286bc4e04c193213b1b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:41:44.254750 10681 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6971716913f14286bc4e04c193213b1b [term 1 LEADER]: Becoming Leader. State: Replica: 6971716913f14286bc4e04c193213b1b, State: Running, Role: LEADER
I20260927 12:41:44.255061 10681 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6971716913f14286bc4e04c193213b1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6971716913f14286bc4e04c193213b1b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43447 } }
I20260927 12:41:44.255618 10676 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6971716913f14286bc4e04c193213b1b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:41:44.256764 10682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6971716913f14286bc4e04c193213b1b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6971716913f14286bc4e04c193213b1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6971716913f14286bc4e04c193213b1b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43447 } } }
I20260927 12:41:44.256774 10683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6971716913f14286bc4e04c193213b1b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6971716913f14286bc4e04c193213b1b. Latest consensus state: current_term: 1 leader_uuid: "6971716913f14286bc4e04c193213b1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6971716913f14286bc4e04c193213b1b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43447 } } }
I20260927 12:41:44.256887 10682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6971716913f14286bc4e04c193213b1b [sys.catalog]: This master's current role is: LEADER
I20260927 12:41:44.256910 10683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6971716913f14286bc4e04c193213b1b [sys.catalog]: This master's current role is: LEADER
I20260927 12:41:44.257313 10690 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:41:44.260648 10690 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:41:44.266101 10690 catalog_manager.cc:1347] Generated new cluster ID: 1ab30009b5314374a75f03e04fe771cd
I20260927 12:41:44.266208 10690 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:41:44.279822 10690 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:41:44.280846 10690 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:41:44.286492 10690 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6971716913f14286bc4e04c193213b1b: Generated new TSK 0
I20260927 12:41:44.287209 10690 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 12:41:44.316963 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43447
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:41:44.430811 10702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:41:44.431077 10702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:41:44.431161 10702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:41:44.435318 10702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:41:44.435385 10702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:41:44.435489 10702 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:41:44.440896 10702 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:41:44.442015 10702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:41:44.443063 10702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:41:44.452978 10710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:44.453225 10702 server_base.cc:1034] running on GCE node
W20260927 12:41:44.452970 10708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:44.453018 10707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:44.453964 10702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:41:44.454674 10702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:41:44.455878 10702 hybrid_clock.cc:648] HybridClock initialized: now 1790512904455863 us; error 40 us; skew 500 ppm
Sep 27 12:41:44 dist-test-slave-bnc1 krb5kdc[10595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512904, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:41:44.459111 10702 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 12:41:44.460127 10702 webserver.cc:460] Webserver started at http://127.0.98.1:44861/ using document root <none> and password file <none>
I20260927 12:41:44.460713 10702 fs_manager.cc:359] Metadata directory not provided
I20260927 12:41:44.460804 10702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:41:44.461041 10702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:41:44.463143 10702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/minicluster-data/ts-0/data/instance:
uuid: "e26007b403f64b44846e858b251cd21a"
format_stamp: "Formatted at 2026-09-27 12:41:44 on dist-test-slave-bnc1"
server_key: "aabf95169f5c1f629e5e9037665e857e"
server_key_iv: "7b65b96e6a199a70099c208343e8c20d"
server_key_version: "encryptionkey@0"
I20260927 12:41:44.463721 10702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e26007b403f64b44846e858b251cd21a"
format_stamp: "Formatted at 2026-09-27 12:41:44 on dist-test-slave-bnc1"
server_key: "aabf95169f5c1f629e5e9037665e857e"
server_key_iv: "7b65b96e6a199a70099c208343e8c20d"
server_key_version: "encryptionkey@0"
I20260927 12:41:44.467415 10702 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20260927 12:41:44.469892 10717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:44.471143 10702 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 12:41:44.471285 10702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/minicluster-data/ts-0/wal
uuid: "e26007b403f64b44846e858b251cd21a"
format_stamp: "Formatted at 2026-09-27 12:41:44 on dist-test-slave-bnc1"
server_key: "aabf95169f5c1f629e5e9037665e857e"
server_key_iv: "7b65b96e6a199a70099c208343e8c20d"
server_key_version: "encryptionkey@0"
I20260927 12:41:44.471423 10702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:41:44.485975 10702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:41:44.488752 10702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:41:44.488971 10702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:41:44.489634 10702 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:41:44.490476 10702 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:41:44.490546 10702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:44.490619 10702 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:41:44.490670 10702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:44.499437 10702 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45083
I20260927 12:41:44.499455 10830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45083 every 8 connection(s)
I20260927 12:41:44.500326 10702 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:41:44.502959 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 10702
I20260927 12:41:44.503085 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/minicluster-data/ts-0/wal/instance
I20260927 12:41:44.503427 392 external_mini_cluster.cc:1428] Setting key 8095bf3cb5763548b474ba1d4c74af54
Sep 27 12:41:44 dist-test-slave-bnc1 krb5kdc[10595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512904, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 12:41:44.515611 10679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:44.502107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44707 (local address 127.0.98.62:43447)
0927 12:41:44.502236 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:44.502239 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:44.503448 (+ 1209us) server_negotiation.cc:407] Connection header received
0927 12:41:44.504504 (+ 1056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:44.504508 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:44.504552 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:44.504642 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:44.506503 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:44.507071 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:44.508656 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:44.509093 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:44.509223 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:44.511907 (+ 2684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:44.511912 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:44.511915 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:41:44.511948 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:44.513665 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:44.514214 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:44.514217 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:44.514218 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:44.514289 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:44.514591 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:44.514593 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:44.514594 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:44.514753 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:44.514867 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:44.515363 (+ 496us) server_negotiation.cc:299] Negotiation successful
0927 12:41:44.515486 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 12:41:44.516696 10833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:44.502470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43447 (local address 127.0.98.1:44707)
0927 12:41:44.503255 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:44.503316 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:44.504300 (+ 984us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:44.504846 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:44.504873 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:44.505408 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:44.506287 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:44.506310 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:44.507203 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:44.507208 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:44.508519 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:44.508529 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:44.509292 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:44.509297 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:44.509473 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:44.510166 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:44.510183 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:44.511730 (+ 1547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:41:44.513772 (+ 2042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:44.513777 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:44.513790 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:44.514103 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:44.514380 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:44.514383 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:44.514384 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:44.514511 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:44.514891 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:41:44.514900 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:41:44.515153 (+ 253us) client_negotiation.cc:769] Sending connection context
0927 12:41:44.515320 (+ 167us) client_negotiation.cc:240] Negotiation successful
0927 12:41:44.515611 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":121,"threads_started":1}
I20260927 12:41:44.518050 10831 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43447
I20260927 12:41:44.518332 10831 heartbeater.cc:464] Registering TS with master...
I20260927 12:41:44.518775 10831 heartbeater.cc:511] Master 127.0.98.62:43447 requested a full tablet report, sending...
I20260927 12:41:44.520294 10640 ts_manager.cc:194] Registered new tserver with Master: e26007b403f64b44846e858b251cd21a (127.0.98.1:45083)
I20260927 12:41:44.521373 10640 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44707
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 12:41:44.536497 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43447
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:41:44.650767 10836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:41:44.651039 10836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:41:44.651113 10836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:41:44.655074 10836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:41:44.655164 10836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:41:44.655249 10836 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:41:44.660600 10836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10836
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:41:44.661710 10836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:41:44.662739 10836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:41:44.672708 10842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:44.672712 10844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:44.672698 10841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:44.672868 10836 server_base.cc:1034] running on GCE node
I20260927 12:41:44.673370 10836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:41:44.674096 10836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:41:44.675278 10836 hybrid_clock.cc:648] HybridClock initialized: now 1790512904675258 us; error 36 us; skew 500 ppm
Sep 27 12:41:44 dist-test-slave-bnc1 krb5kdc[10595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512904, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:41:44.678025 10836 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 12:41:44.678871 10836 webserver.cc:460] Webserver started at http://127.0.98.2:42355/ using document root <none> and password file <none>
I20260927 12:41:44.679426 10836 fs_manager.cc:359] Metadata directory not provided
I20260927 12:41:44.679515 10836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:41:44.679731 10836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:41:44.681859 10836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/minicluster-data/ts-1/data/instance:
uuid: "ee51b18c18e4447ca61124e81521b7a6"
format_stamp: "Formatted at 2026-09-27 12:41:44 on dist-test-slave-bnc1"
server_key: "449b506c5b64104bb5add71a9390be7b"
server_key_iv: "bbc994984a9167bf0ffdbb5ad654de86"
server_key_version: "encryptionkey@0"
I20260927 12:41:44.682417 10836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ee51b18c18e4447ca61124e81521b7a6"
format_stamp: "Formatted at 2026-09-27 12:41:44 on dist-test-slave-bnc1"
server_key: "449b506c5b64104bb5add71a9390be7b"
server_key_iv: "bbc994984a9167bf0ffdbb5ad654de86"
server_key_version: "encryptionkey@0"
I20260927 12:41:44.685997 10836 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:41:44.688261 10851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:44.689441 10836 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:41:44.689608 10836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/minicluster-data/ts-1/wal
uuid: "ee51b18c18e4447ca61124e81521b7a6"
format_stamp: "Formatted at 2026-09-27 12:41:44 on dist-test-slave-bnc1"
server_key: "449b506c5b64104bb5add71a9390be7b"
server_key_iv: "bbc994984a9167bf0ffdbb5ad654de86"
server_key_version: "encryptionkey@0"
I20260927 12:41:44.689750 10836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:41:44.723410 10836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:41:44.726435 10836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:41:44.726652 10836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:41:44.727281 10836 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:41:44.728132 10836 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:41:44.728204 10836 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:44.728276 10836 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:41:44.728327 10836 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:44.737406 10836 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35467
I20260927 12:41:44.737428 10964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35467 every 8 connection(s)
I20260927 12:41:44.738345 10836 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:41:44.744011 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 10836
I20260927 12:41:44.744112 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/minicluster-data/ts-1/wal/instance
I20260927 12:41:44.744464 392 external_mini_cluster.cc:1428] Setting key 6eb17a46714e3a619f87fd30b9ba9451
Sep 27 12:41:44 dist-test-slave-bnc1 krb5kdc[10595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512904, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:41:44.752836 10679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:44.740188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41077 (local address 127.0.98.62:43447)
0927 12:41:44.740339 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:44.740343 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:44.741428 (+ 1085us) server_negotiation.cc:407] Connection header received
0927 12:41:44.742339 (+ 911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:44.742342 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:44.742391 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:44.742472 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:44.743605 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:44.744246 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:44.745775 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:44.746058 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:44.746142 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:44.749014 (+ 2872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:44.749019 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:44.749022 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:41:44.749055 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:44.750548 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:44.751189 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:44.751193 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:44.751195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:44.751283 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:44.751694 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:44.751696 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:44.751697 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:44.751869 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:44.751951 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:44.752621 (+ 670us) server_negotiation.cc:299] Negotiation successful
0927 12:41:44.752715 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 12:41:44.754151 10967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:44.740622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43447 (local address 127.0.98.2:41077)
0927 12:41:44.741274 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:44.741314 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:44.742150 (+ 836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:44.742645 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:44.742652 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:44.743008 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:44.743474 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:44.743486 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:44.744379 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:44.744383 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:44.745632 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:44.745643 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:44.746209 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:44.746214 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:44.746437 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:44.747082 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:44.747108 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:44.748807 (+ 1699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:41:44.750663 (+ 1856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:44.750672 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:44.750693 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:44.751083 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:44.751376 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:44.751382 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:44.751387 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:44.751599 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:44.751972 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:41:44.751981 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:41:44.752345 (+ 364us) client_negotiation.cc:769] Sending connection context
0927 12:41:44.752589 (+ 244us) client_negotiation.cc:240] Negotiation successful
0927 12:41:44.752968 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":132,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 12:41:44.755330 10965 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43447
I20260927 12:41:44.755627 10965 heartbeater.cc:464] Registering TS with master...
I20260927 12:41:44.756116 10965 heartbeater.cc:511] Master 127.0.98.62:43447 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 12:41:44.757296 10640 ts_manager.cc:194] Registered new tserver with Master: ee51b18c18e4447ca61124e81521b7a6 (127.0.98.2:35467)
I20260927 12:41:44.758177 10640 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41077
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 12:41:44.770746 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43447
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:41:44.879082 10970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:41:44.879384 10970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:41:44.879457 10970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:41:44.883512 10970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:41:44.883599 10970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:41:44.883683 10970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:41:44.888957 10970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:41:44.890108 10970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:41:44.891146 10970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 12:41:44.900934 10970 server_base.cc:1034] running on GCE node
W20260927 12:41:44.901142 10976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:44.901126 10975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:41:44.901126 10978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:41:44.901763 10970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:41:44.902451 10970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:41:44.903664 10970 hybrid_clock.cc:648] HybridClock initialized: now 1790512904903652 us; error 33 us; skew 500 ppm
Sep 27 12:41:44 dist-test-slave-bnc1 krb5kdc[10595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512904, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:41:44.906827 10970 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 12:41:44.907752 10970 webserver.cc:460] Webserver started at http://127.0.98.3:44557/ using document root <none> and password file <none>
I20260927 12:41:44.908332 10970 fs_manager.cc:359] Metadata directory not provided
I20260927 12:41:44.908434 10970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:41:44.908665 10970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:41:44.910857 10970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/minicluster-data/ts-2/data/instance:
uuid: "feef156dd078440cb1eb8fe78e845b8d"
format_stamp: "Formatted at 2026-09-27 12:41:44 on dist-test-slave-bnc1"
server_key: "6355662dab3102533a6297ee3c687a7f"
server_key_iv: "8f9770b7074c4890220493da7448c418"
server_key_version: "encryptionkey@0"
I20260927 12:41:44.911489 10970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/minicluster-data/ts-2/wal/instance:
uuid: "feef156dd078440cb1eb8fe78e845b8d"
format_stamp: "Formatted at 2026-09-27 12:41:44 on dist-test-slave-bnc1"
server_key: "6355662dab3102533a6297ee3c687a7f"
server_key_iv: "8f9770b7074c4890220493da7448c418"
server_key_version: "encryptionkey@0"
I20260927 12:41:44.915200 10970 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 12:41:44.917649 10985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:44.918941 10970 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 12:41:44.919073 10970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/minicluster-data/ts-2/wal
uuid: "feef156dd078440cb1eb8fe78e845b8d"
format_stamp: "Formatted at 2026-09-27 12:41:44 on dist-test-slave-bnc1"
server_key: "6355662dab3102533a6297ee3c687a7f"
server_key_iv: "8f9770b7074c4890220493da7448c418"
server_key_version: "encryptionkey@0"
I20260927 12:41:44.919179 10970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:41:44.931321 10970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:41:44.934821 10970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:41:44.935032 10970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:41:44.935635 10970 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:41:44.936487 10970 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:41:44.936555 10970 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:44.936618 10970 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:41:44.936658 10970 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:41:44.945109 10970 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40753
I20260927 12:41:44.945142 11098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40753 every 8 connection(s)
I20260927 12:41:44.945986 10970 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:41:44.946192 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 10970
I20260927 12:41:44.946333 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/minicluster-data/ts-2/wal/instance
I20260927 12:41:44.946667 392 external_mini_cluster.cc:1428] Setting key 497f4c07811b28791048bdc416425055
Sep 27 12:41:44 dist-test-slave-bnc1 krb5kdc[10595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512904, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:41:44.959070 10679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:44.947763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48841 (local address 127.0.98.62:43447)
0927 12:41:44.947904 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:44.947909 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:44.948787 (+ 878us) server_negotiation.cc:407] Connection header received
0927 12:41:44.950116 (+ 1329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:44.950118 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:44.950163 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:44.950232 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:44.951320 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:44.951839 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:44.952933 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:44.953136 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:44.953213 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:44.955597 (+ 2384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:44.955600 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:44.955602 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:41:44.955628 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:44.957225 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:44.957779 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:44.957782 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:44.957783 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:44.957853 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:44.958171 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:44.958173 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:44.958174 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:44.958339 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:44.958412 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:44.958849 (+ 437us) server_negotiation.cc:299] Negotiation successful
0927 12:41:44.958940 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 12:41:44.960170 11101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:44.948229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43447 (local address 127.0.98.3:48841)
0927 12:41:44.948656 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:44.948705 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:44.949929 (+ 1224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:44.950388 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:44.950395 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:44.950737 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:41:44.951202 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:44.951215 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:44.951937 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:44.951939 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:44.952833 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:44.952841 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:44.953255 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:44.953258 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:44.953396 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:44.953950 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:41:44.953966 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:41:44.955443 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:41:44.957324 (+ 1881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:44.957328 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:44.957340 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:44.957673 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:44.957956 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:41:44.957958 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:41:44.957959 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:41:44.958091 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:41:44.958420 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:41:44.958425 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:41:44.958651 (+ 226us) client_negotiation.cc:769] Sending connection context
0927 12:41:44.958795 (+ 144us) client_negotiation.cc:240] Negotiation successful
0927 12:41:44.959176 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":127,"threads_started":1}
I20260927 12:41:44.961277 11099 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43447
I20260927 12:41:44.961524 11099 heartbeater.cc:464] Registering TS with master...
I20260927 12:41:44.962040 11099 heartbeater.cc:511] Master 127.0.98.62:43447 requested a full tablet report, sending...
I20260927 12:41:44.962970 10640 ts_manager.cc:194] Registered new tserver with Master: feef156dd078440cb1eb8fe78e845b8d (127.0.98.3:40753)
I20260927 12:41:44.963517 10640 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48841
I20260927 12:41:44.972107 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 12:41:44.981441 10679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:44.974019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33086 (local address 127.0.98.62:43447)
0927 12:41:44.974176 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:44.974181 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:44.974379 (+ 198us) server_negotiation.cc:407] Connection header received
0927 12:41:44.974545 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:44.974548 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:44.974598 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:44.974682 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:41:44.975479 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:44.976008 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:44.976857 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:44.977092 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:44.977202 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:44.978556 (+ 1354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:41:44.978560 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:41:44.978562 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:41:44.978602 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:41:44.979816 (+ 1214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:44.980215 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:44.980219 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:44.980221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:44.980299 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:41:44.980585 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:41:44.980588 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:41:44.980590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:41:44.980802 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:41:44.980937 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:44.981167 (+ 230us) server_negotiation.cc:299] Negotiation successful
0927 12:41:44.981279 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 12:41:44.984133 10640 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33086:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 12:41:44.987098 10640 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:41:45.001704 11115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:44.996770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60594 (local address 127.0.98.3:40753)
0927 12:41:44.997236 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:44.997247 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:44.997422 (+ 175us) server_negotiation.cc:407] Connection header received
0927 12:41:44.997526 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:44.997530 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:44.997725 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:44.997876 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:41:44.998084 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:44.999299 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:45.000396 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:45.001001 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:45.001127 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:45.001274 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:45.001463 (+ 189us) server_negotiation.cc:299] Negotiation successful
0927 12:41:45.001551 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":167,"threads_started":1}
I20260927 12:41:45.001698 11114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:44.996601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40753 (local address 127.0.0.1:60594)
0927 12:41:44.997317 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:44.997329 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:44.997426 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:44.997868 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:44.997871 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:44.997888 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:41:44.997979 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:44.997988 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:44.999418 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:44.999424 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:45.000272 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:45.000281 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:45.001126 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:45.001130 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:45.001230 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:45.001268 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 12:41:45.001391 (+ 123us) client_negotiation.cc:240] Negotiation successful
0927 12:41:45.001514 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"mutex_wait_us":1,"thread_start_us":85,"threads_started":1}
I20260927 12:41:45.002378 11111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:44.995580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46306 (local address 127.0.98.1:45083)
0927 12:41:44.996129 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:44.996148 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:44.996176 (+ 28us) server_negotiation.cc:407] Connection header received
0927 12:41:44.996254 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:44.996259 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:44.996488 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:44.996647 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:41:44.997322 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:44.998605 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:45.000363 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:45.001189 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:45.001300 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:45.001443 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:45.002109 (+ 666us) server_negotiation.cc:299] Negotiation successful
0927 12:41:45.002219 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":98,"threads_started":1}
I20260927 12:41:45.002368 11110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:44.995446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45083 (local address 127.0.0.1:46306)
0927 12:41:44.995796 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:44.995849 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:44.996009 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:44.996865 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:44.996871 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:44.996894 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:41:44.997180 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:44.997191 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:44.998755 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:44.998758 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:45.000234 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:45.000244 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:45.001503 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:45.001507 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:45.001662 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:45.001685 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 12:41:45.001774 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 12:41:45.001862 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":25,"thread_start_us":82,"threads_started":1}
I20260927 12:41:45.002367 11112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:44.996249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35467 (local address 127.0.0.1:44476)
0927 12:41:44.996878 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:44.996890 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:44.997055 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:44.997650 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:44.997653 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:44.997671 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:41:44.997765 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:44.997770 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:44.999160 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:44.999163 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:45.000390 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:45.000398 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:45.001503 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:45.001507 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:45.002042 (+ 535us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:45.002074 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 12:41:45.002160 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 12:41:45.002237 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"mutex_wait_us":407,"thread_start_us":101,"threads_started":1}
I20260927 12:41:45.002709 11113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:44.996396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44476 (local address 127.0.98.2:35467)
0927 12:41:44.996827 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:44.996842 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:44.996979 (+ 137us) server_negotiation.cc:407] Connection header received
0927 12:41:44.997228 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:44.997235 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:44.997424 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:44.997569 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:41:44.997894 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:44.999013 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:45.000478 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:45.001222 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:45.001311 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:45.001443 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:45.002436 (+ 993us) server_negotiation.cc:299] Negotiation successful
0927 12:41:45.002548 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":153,"threads_started":1}
I20260927 12:41:45.004819 10765 tablet_service.cc:1467] Processing CreateTablet for tablet 37f035855cfb4d51a55be3f08e9b314e (DEFAULT_TABLE table=test-table [id=29420e005b224ddc9e43f35dc644686f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:41:45.004940 11033 tablet_service.cc:1467] Processing CreateTablet for tablet 37f035855cfb4d51a55be3f08e9b314e (DEFAULT_TABLE table=test-table [id=29420e005b224ddc9e43f35dc644686f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:41:45.004940 10899 tablet_service.cc:1467] Processing CreateTablet for tablet 37f035855cfb4d51a55be3f08e9b314e (DEFAULT_TABLE table=test-table [id=29420e005b224ddc9e43f35dc644686f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:41:45.005949 10899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37f035855cfb4d51a55be3f08e9b314e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:41:45.005954 11033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37f035855cfb4d51a55be3f08e9b314e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:41:45.005949 10765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37f035855cfb4d51a55be3f08e9b314e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:41:45.013656 11117 tablet_bootstrap.cc:492] T 37f035855cfb4d51a55be3f08e9b314e P ee51b18c18e4447ca61124e81521b7a6: Bootstrap starting.
I20260927 12:41:45.014130 11118 tablet_bootstrap.cc:492] T 37f035855cfb4d51a55be3f08e9b314e P feef156dd078440cb1eb8fe78e845b8d: Bootstrap starting.
I20260927 12:41:45.014245 11116 tablet_bootstrap.cc:492] T 37f035855cfb4d51a55be3f08e9b314e P e26007b403f64b44846e858b251cd21a: Bootstrap starting.
I20260927 12:41:45.016059 11116 tablet_bootstrap.cc:654] T 37f035855cfb4d51a55be3f08e9b314e P e26007b403f64b44846e858b251cd21a: Neither blocks nor log segments found. Creating new log.
I20260927 12:41:45.016532 11117 tablet_bootstrap.cc:654] T 37f035855cfb4d51a55be3f08e9b314e P ee51b18c18e4447ca61124e81521b7a6: Neither blocks nor log segments found. Creating new log.
I20260927 12:41:45.016595 11118 tablet_bootstrap.cc:654] T 37f035855cfb4d51a55be3f08e9b314e P feef156dd078440cb1eb8fe78e845b8d: Neither blocks nor log segments found. Creating new log.
I20260927 12:41:45.017355 11116 log.cc:824] T 37f035855cfb4d51a55be3f08e9b314e P e26007b403f64b44846e858b251cd21a: Log is configured to *not* fsync() on all Append() calls
I20260927 12:41:45.017406 11117 log.cc:824] T 37f035855cfb4d51a55be3f08e9b314e P ee51b18c18e4447ca61124e81521b7a6: Log is configured to *not* fsync() on all Append() calls
I20260927 12:41:45.017534 11118 log.cc:824] T 37f035855cfb4d51a55be3f08e9b314e P feef156dd078440cb1eb8fe78e845b8d: Log is configured to *not* fsync() on all Append() calls
I20260927 12:41:45.019207 11118 tablet_bootstrap.cc:492] T 37f035855cfb4d51a55be3f08e9b314e P feef156dd078440cb1eb8fe78e845b8d: No bootstrap required, opened a new log
I20260927 12:41:45.019239 11116 tablet_bootstrap.cc:492] T 37f035855cfb4d51a55be3f08e9b314e P e26007b403f64b44846e858b251cd21a: No bootstrap required, opened a new log
I20260927 12:41:45.019409 11118 ts_tablet_manager.cc:1397] T 37f035855cfb4d51a55be3f08e9b314e P feef156dd078440cb1eb8fe78e845b8d: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260927 12:41:45.019430 11116 ts_tablet_manager.cc:1397] T 37f035855cfb4d51a55be3f08e9b314e P e26007b403f64b44846e858b251cd21a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 12:41:45.020428 11117 tablet_bootstrap.cc:492] T 37f035855cfb4d51a55be3f08e9b314e P ee51b18c18e4447ca61124e81521b7a6: No bootstrap required, opened a new log
I20260927 12:41:45.020588 11117 ts_tablet_manager.cc:1397] T 37f035855cfb4d51a55be3f08e9b314e P ee51b18c18e4447ca61124e81521b7a6: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260927 12:41:45.022318 11118 raft_consensus.cc:348] T 37f035855cfb4d51a55be3f08e9b314e P feef156dd078440cb1eb8fe78e845b8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26007b403f64b44846e858b251cd21a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45083 } } peers { permanent_uuid: "ee51b18c18e4447ca61124e81521b7a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35467 } } peers { permanent_uuid: "feef156dd078440cb1eb8fe78e845b8d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40753 } }
I20260927 12:41:45.022600 11118 raft_consensus.cc:374] T 37f035855cfb4d51a55be3f08e9b314e P feef156dd078440cb1eb8fe78e845b8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:41:45.022655 11118 raft_consensus.cc:729] T 37f035855cfb4d51a55be3f08e9b314e P feef156dd078440cb1eb8fe78e845b8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feef156dd078440cb1eb8fe78e845b8d, State: Initialized, Role: FOLLOWER
I20260927 12:41:45.023098 11118 consensus_queue.cc:260] T 37f035855cfb4d51a55be3f08e9b314e P feef156dd078440cb1eb8fe78e845b8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26007b403f64b44846e858b251cd21a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45083 } } peers { permanent_uuid: "ee51b18c18e4447ca61124e81521b7a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35467 } } peers { permanent_uuid: "feef156dd078440cb1eb8fe78e845b8d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40753 } }
I20260927 12:41:45.023129 11116 raft_consensus.cc:348] T 37f035855cfb4d51a55be3f08e9b314e P e26007b403f64b44846e858b251cd21a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26007b403f64b44846e858b251cd21a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45083 } } peers { permanent_uuid: "ee51b18c18e4447ca61124e81521b7a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35467 } } peers { permanent_uuid: "feef156dd078440cb1eb8fe78e845b8d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40753 } }
I20260927 12:41:45.023365 11116 raft_consensus.cc:374] T 37f035855cfb4d51a55be3f08e9b314e P e26007b403f64b44846e858b251cd21a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:41:45.023433 11116 raft_consensus.cc:729] T 37f035855cfb4d51a55be3f08e9b314e P e26007b403f64b44846e858b251cd21a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e26007b403f64b44846e858b251cd21a, State: Initialized, Role: FOLLOWER
I20260927 12:41:45.023852 11116 consensus_queue.cc:260] T 37f035855cfb4d51a55be3f08e9b314e P e26007b403f64b44846e858b251cd21a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26007b403f64b44846e858b251cd21a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45083 } } peers { permanent_uuid: "ee51b18c18e4447ca61124e81521b7a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35467 } } peers { permanent_uuid: "feef156dd078440cb1eb8fe78e845b8d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40753 } }
I20260927 12:41:45.024101 11118 ts_tablet_manager.cc:1428] T 37f035855cfb4d51a55be3f08e9b314e P feef156dd078440cb1eb8fe78e845b8d: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260927 12:41:45.024434 11099 heartbeater.cc:503] Master 127.0.98.62:43447 was elected leader, sending a full tablet report...
I20260927 12:41:45.024775 11116 ts_tablet_manager.cc:1428] T 37f035855cfb4d51a55be3f08e9b314e P e26007b403f64b44846e858b251cd21a: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 12:41:45.024680 11117 raft_consensus.cc:348] T 37f035855cfb4d51a55be3f08e9b314e P ee51b18c18e4447ca61124e81521b7a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26007b403f64b44846e858b251cd21a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45083 } } peers { permanent_uuid: "ee51b18c18e4447ca61124e81521b7a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35467 } } peers { permanent_uuid: "feef156dd078440cb1eb8fe78e845b8d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40753 } }
I20260927 12:41:45.024941 11117 raft_consensus.cc:374] T 37f035855cfb4d51a55be3f08e9b314e P ee51b18c18e4447ca61124e81521b7a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:41:45.024997 11117 raft_consensus.cc:729] T 37f035855cfb4d51a55be3f08e9b314e P ee51b18c18e4447ca61124e81521b7a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee51b18c18e4447ca61124e81521b7a6, State: Initialized, Role: FOLLOWER
I20260927 12:41:45.025098 10831 heartbeater.cc:503] Master 127.0.98.62:43447 was elected leader, sending a full tablet report...
I20260927 12:41:45.025398 11117 consensus_queue.cc:260] T 37f035855cfb4d51a55be3f08e9b314e P ee51b18c18e4447ca61124e81521b7a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26007b403f64b44846e858b251cd21a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45083 } } peers { permanent_uuid: "ee51b18c18e4447ca61124e81521b7a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35467 } } peers { permanent_uuid: "feef156dd078440cb1eb8fe78e845b8d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40753 } }
I20260927 12:41:45.026055 10965 heartbeater.cc:503] Master 127.0.98.62:43447 was elected leader, sending a full tablet report...
I20260927 12:41:45.026338 11117 ts_tablet_manager.cc:1428] T 37f035855cfb4d51a55be3f08e9b314e P ee51b18c18e4447ca61124e81521b7a6: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260927 12:41:45.196951 11100 tablet.cc:2369] T 37f035855cfb4d51a55be3f08e9b314e P feef156dd078440cb1eb8fe78e845b8d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:41:45.208674 11123 raft_consensus.cc:482] T 37f035855cfb4d51a55be3f08e9b314e P e26007b403f64b44846e858b251cd21a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:41:45.208879 11123 raft_consensus.cc:504] T 37f035855cfb4d51a55be3f08e9b314e P e26007b403f64b44846e858b251cd21a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26007b403f64b44846e858b251cd21a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45083 } } peers { permanent_uuid: "ee51b18c18e4447ca61124e81521b7a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35467 } } peers { permanent_uuid: "feef156dd078440cb1eb8fe78e845b8d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40753 } }
I20260927 12:41:45.209019 11124 raft_consensus.cc:482] T 37f035855cfb4d51a55be3f08e9b314e P ee51b18c18e4447ca61124e81521b7a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:41:45.209221 11124 raft_consensus.cc:504] T 37f035855cfb4d51a55be3f08e9b314e P ee51b18c18e4447ca61124e81521b7a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26007b403f64b44846e858b251cd21a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45083 } } peers { permanent_uuid: "ee51b18c18e4447ca61124e81521b7a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35467 } } peers { permanent_uuid: "feef156dd078440cb1eb8fe78e845b8d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40753 } }
I20260927 12:41:45.210259 11123 leader_election.cc:288] T 37f035855cfb4d51a55be3f08e9b314e P e26007b403f64b44846e858b251cd21a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee51b18c18e4447ca61124e81521b7a6 (127.0.98.2:35467), feef156dd078440cb1eb8fe78e845b8d (127.0.98.3:40753)
I20260927 12:41:45.210367 11124 leader_election.cc:288] T 37f035855cfb4d51a55be3f08e9b314e P ee51b18c18e4447ca61124e81521b7a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e26007b403f64b44846e858b251cd21a (127.0.98.1:45083), feef156dd078440cb1eb8fe78e845b8d (127.0.98.3:40753)
I20260927 12:41:45.215059 10967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:45.210538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45083 (local address 127.0.98.2:47367)
0927 12:41:45.210847 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:45.210865 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:45.210987 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:45.211285 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:45.211288 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:45.211306 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:41:45.211429 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:45.211436 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:45.212421 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:45.212425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:45.213671 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:45.213680 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:45.214658 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:45.214664 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:45.214773 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:45.214797 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 12:41:45.214861 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 12:41:45.214926 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246}
I20260927 12:41:45.215487 11111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:45.210621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47367 (local address 127.0.98.1:45083)
0927 12:41:45.210748 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:45.210753 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:45.211085 (+ 332us) server_negotiation.cc:407] Connection header received
0927 12:41:45.211140 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:45.211143 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:45.211179 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:45.211264 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:41:45.211543 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:45.212300 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:45.213799 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:45.214445 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:45.214536 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:45.214604 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:45.215300 (+ 696us) server_negotiation.cc:299] Negotiation successful
0927 12:41:45.215368 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":40}
I20260927 12:41:45.215991 11127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:45.210887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40753 (local address 127.0.98.1:40351)
0927 12:41:45.211576 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:45.211589 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:45.211681 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:45.212222 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:45.212227 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:45.212243 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:41:45.212336 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:45.212342 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:45.213257 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:45.213260 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:45.214488 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:45.214494 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:45.215441 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:45.215444 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:45.215642 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:45.215705 (+ 63us) client_negotiation.cc:769] Sending connection context
0927 12:41:45.215778 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 12:41:45.215847 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"mutex_wait_us":143,"thread_start_us":99,"threads_started":1}
I20260927 12:41:45.215991 11113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:45.210523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38581 (local address 127.0.98.2:35467)
0927 12:41:45.210654 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:45.210659 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:45.211555 (+ 896us) server_negotiation.cc:407] Connection header received
0927 12:41:45.211749 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:45.211753 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:45.211796 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:45.211878 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:41:45.212170 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:45.212949 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:45.214617 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:45.215215 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:45.215297 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:45.215344 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:45.215823 (+ 479us) server_negotiation.cc:299] Negotiation successful
0927 12:41:45.215883 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":45}
I20260927 12:41:45.216017 10785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37f035855cfb4d51a55be3f08e9b314e" candidate_uuid: "ee51b18c18e4447ca61124e81521b7a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e26007b403f64b44846e858b251cd21a" is_pre_election: true
I20260927 12:41:45.216229 10785 raft_consensus.cc:2454] T 37f035855cfb4d51a55be3f08e9b314e P e26007b403f64b44846e858b251cd21a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee51b18c18e4447ca61124e81521b7a6 in term 0.
I20260927 12:41:45.216331 11115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:45.211003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40351 (local address 127.0.98.3:40753)
0927 12:41:45.211142 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:45.211146 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:45.211682 (+ 536us) server_negotiation.cc:407] Connection header received
0927 12:41:45.211988 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:45.211991 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:45.212040 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:45.212113 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:41:45.212452 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:45.213148 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:45.214617 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:45.215245 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:45.215309 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:45.215357 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:45.216136 (+ 779us) server_negotiation.cc:299] Negotiation successful
0927 12:41:45.216203 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":50}
I20260927 12:41:45.216387 11125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:45.210472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35467 (local address 127.0.98.1:38581)
0927 12:41:45.211480 (+ 1008us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:45.211494 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:45.211606 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:45.211887 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:45.211890 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:45.211905 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:41:45.212035 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:45.212041 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:45.213081 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:45.213084 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:45.214478 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:45.214488 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:45.215445 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:45.215447 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:45.215617 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:45.215708 (+ 91us) client_negotiation.cc:769] Sending connection context
0927 12:41:45.216151 (+ 443us) client_negotiation.cc:240] Negotiation successful
0927 12:41:45.216212 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":942,"mutex_wait_us":108,"thread_start_us":131,"threads_started":1}
I20260927 12:41:45.216816 11126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:45.210791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40753 (local address 127.0.98.2:46533)
0927 12:41:45.211254 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:45.211270 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 12:41:45.211374 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:41:45.212161 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:41:45.212163 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:41:45.212179 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:41:45.212321 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:45.212330 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:45.213416 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:45.213419 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:45.214600 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:41:45.214609 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:41:45.215910 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:41:45.215913 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:41:45.215995 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:45.216020 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 12:41:45.216610 (+ 590us) client_negotiation.cc:240] Negotiation successful
0927 12:41:45.216693 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"mutex_wait_us":29,"thread_start_us":107,"threads_started":1}
I20260927 12:41:45.216933 11053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37f035855cfb4d51a55be3f08e9b314e" candidate_uuid: "e26007b403f64b44846e858b251cd21a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feef156dd078440cb1eb8fe78e845b8d" is_pre_election: true
I20260927 12:41:45.217196 11053 raft_consensus.cc:2454] T 37f035855cfb4d51a55be3f08e9b314e P feef156dd078440cb1eb8fe78e845b8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e26007b403f64b44846e858b251cd21a in term 0.
I20260927 12:41:45.217243 11128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:45.211212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46533 (local address 127.0.98.3:40753)
0927 12:41:45.211860 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:45.211868 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:45.211880 (+ 12us) server_negotiation.cc:407] Connection header received
0927 12:41:45.211924 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:45.211927 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:45.211967 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:45.212040 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:41:45.212439 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:45.213290 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:45.214720 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:45.215686 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:45.215781 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:45.215842 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:45.217027 (+ 1185us) server_negotiation.cc:299] Negotiation successful
0927 12:41:45.217125 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":208,"server-negotiator.queue_time_us":585,"thread_start_us":91,"threads_started":1}
I20260927 12:41:45.216842 10853 leader_election.cc:302] T 37f035855cfb4d51a55be3f08e9b314e P ee51b18c18e4447ca61124e81521b7a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e26007b403f64b44846e858b251cd21a, ee51b18c18e4447ca61124e81521b7a6; no voters:
I20260927 12:41:45.216930 10919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37f035855cfb4d51a55be3f08e9b314e" candidate_uuid: "e26007b403f64b44846e858b251cd21a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee51b18c18e4447ca61124e81521b7a6" is_pre_election: true
I20260927 12:41:45.217626 10919 raft_consensus.cc:2454] T 37f035855cfb4d51a55be3f08e9b314e P ee51b18c18e4447ca61124e81521b7a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e26007b403f64b44846e858b251cd21a in term 0.
I20260927 12:41:45.217617 10719 leader_election.cc:302] T 37f035855cfb4d51a55be3f08e9b314e P e26007b403f64b44846e858b251cd21a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e26007b403f64b44846e858b251cd21a, feef156dd078440cb1eb8fe78e845b8d; no voters:
I20260927 12:41:45.217886 11124 raft_consensus.cc:2781] T 37f035855cfb4d51a55be3f08e9b314e P ee51b18c18e4447ca61124e81521b7a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:41:45.217911 11123 raft_consensus.cc:2781] T 37f035855cfb4d51a55be3f08e9b314e P e26007b403f64b44846e858b251cd21a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:41:45.217942 11053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37f035855cfb4d51a55be3f08e9b314e" candidate_uuid: "ee51b18c18e4447ca61124e81521b7a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feef156dd078440cb1eb8fe78e845b8d" is_pre_election: true
I20260927 12:41:45.217981 11124 raft_consensus.cc:482] T 37f035855cfb4d51a55be3f08e9b314e P ee51b18c18e4447ca61124e81521b7a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:41:45.217990 11123 raft_consensus.cc:482] T 37f035855cfb4d51a55be3f08e9b314e P e26007b403f64b44846e858b251cd21a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:41:45.218029 11124 raft_consensus.cc:3037] T 37f035855cfb4d51a55be3f08e9b314e P ee51b18c18e4447ca61124e81521b7a6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:41:45.218034 11123 raft_consensus.cc:3037] T 37f035855cfb4d51a55be3f08e9b314e P e26007b403f64b44846e858b251cd21a [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:41:45.218051 11053 raft_consensus.cc:2454] T 37f035855cfb4d51a55be3f08e9b314e P feef156dd078440cb1eb8fe78e845b8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee51b18c18e4447ca61124e81521b7a6 in term 0.
I20260927 12:41:45.219044 11123 raft_consensus.cc:504] T 37f035855cfb4d51a55be3f08e9b314e P e26007b403f64b44846e858b251cd21a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26007b403f64b44846e858b251cd21a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45083 } } peers { permanent_uuid: "ee51b18c18e4447ca61124e81521b7a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35467 } } peers { permanent_uuid: "feef156dd078440cb1eb8fe78e845b8d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40753 } }
I20260927 12:41:45.219146 11124 raft_consensus.cc:504] T 37f035855cfb4d51a55be3f08e9b314e P ee51b18c18e4447ca61124e81521b7a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26007b403f64b44846e858b251cd21a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45083 } } peers { permanent_uuid: "ee51b18c18e4447ca61124e81521b7a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35467 } } peers { permanent_uuid: "feef156dd078440cb1eb8fe78e845b8d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40753 } }
I20260927 12:41:45.219429 11123 leader_election.cc:288] T 37f035855cfb4d51a55be3f08e9b314e P e26007b403f64b44846e858b251cd21a [CANDIDATE]: Term 1 election: Requested vote from peers ee51b18c18e4447ca61124e81521b7a6 (127.0.98.2:35467), feef156dd078440cb1eb8fe78e845b8d (127.0.98.3:40753)
I20260927 12:41:45.219523 11124 leader_election.cc:288] T 37f035855cfb4d51a55be3f08e9b314e P ee51b18c18e4447ca61124e81521b7a6 [CANDIDATE]: Term 1 election: Requested vote from peers e26007b403f64b44846e858b251cd21a (127.0.98.1:45083), feef156dd078440cb1eb8fe78e845b8d (127.0.98.3:40753)
I20260927 12:41:45.219738 10919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37f035855cfb4d51a55be3f08e9b314e" candidate_uuid: "e26007b403f64b44846e858b251cd21a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee51b18c18e4447ca61124e81521b7a6"
I20260927 12:41:45.219817 11053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37f035855cfb4d51a55be3f08e9b314e" candidate_uuid: "e26007b403f64b44846e858b251cd21a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feef156dd078440cb1eb8fe78e845b8d"
I20260927 12:41:45.219842 11052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37f035855cfb4d51a55be3f08e9b314e" candidate_uuid: "ee51b18c18e4447ca61124e81521b7a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feef156dd078440cb1eb8fe78e845b8d"
I20260927 12:41:45.219897 10919 raft_consensus.cc:2379] T 37f035855cfb4d51a55be3f08e9b314e P ee51b18c18e4447ca61124e81521b7a6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate e26007b403f64b44846e858b251cd21a in current term 1: Already voted for candidate ee51b18c18e4447ca61124e81521b7a6 in this term.
I20260927 12:41:45.219969 10785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37f035855cfb4d51a55be3f08e9b314e" candidate_uuid: "ee51b18c18e4447ca61124e81521b7a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e26007b403f64b44846e858b251cd21a"
I20260927 12:41:45.219969 11053 raft_consensus.cc:3037] T 37f035855cfb4d51a55be3f08e9b314e P feef156dd078440cb1eb8fe78e845b8d [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:41:45.220124 10785 raft_consensus.cc:2379] T 37f035855cfb4d51a55be3f08e9b314e P e26007b403f64b44846e858b251cd21a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ee51b18c18e4447ca61124e81521b7a6 in current term 1: Already voted for candidate e26007b403f64b44846e858b251cd21a in this term.
I20260927 12:41:45.220526 10853 leader_election.cc:302] T 37f035855cfb4d51a55be3f08e9b314e P ee51b18c18e4447ca61124e81521b7a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee51b18c18e4447ca61124e81521b7a6; no voters: e26007b403f64b44846e858b251cd21a, feef156dd078440cb1eb8fe78e845b8d
I20260927 12:41:45.220669 11124 raft_consensus.cc:2726] T 37f035855cfb4d51a55be3f08e9b314e P ee51b18c18e4447ca61124e81521b7a6 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260927 12:41:45.221149 11053 raft_consensus.cc:2454] T 37f035855cfb4d51a55be3f08e9b314e P feef156dd078440cb1eb8fe78e845b8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e26007b403f64b44846e858b251cd21a in term 1.
I20260927 12:41:45.221418 10719 leader_election.cc:302] T 37f035855cfb4d51a55be3f08e9b314e P e26007b403f64b44846e858b251cd21a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e26007b403f64b44846e858b251cd21a, feef156dd078440cb1eb8fe78e845b8d; no voters: ee51b18c18e4447ca61124e81521b7a6
I20260927 12:41:45.221606 11123 raft_consensus.cc:2781] T 37f035855cfb4d51a55be3f08e9b314e P e26007b403f64b44846e858b251cd21a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:41:45.221800 11123 raft_consensus.cc:686] T 37f035855cfb4d51a55be3f08e9b314e P e26007b403f64b44846e858b251cd21a [term 1 LEADER]: Becoming Leader. State: Replica: e26007b403f64b44846e858b251cd21a, State: Running, Role: LEADER
I20260927 12:41:45.222100 11123 consensus_queue.cc:237] T 37f035855cfb4d51a55be3f08e9b314e P e26007b403f64b44846e858b251cd21a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26007b403f64b44846e858b251cd21a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45083 } } peers { permanent_uuid: "ee51b18c18e4447ca61124e81521b7a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35467 } } peers { permanent_uuid: "feef156dd078440cb1eb8fe78e845b8d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40753 } }
I20260927 12:41:45.225438 10640 catalog_manager.cc:5495] T 37f035855cfb4d51a55be3f08e9b314e P e26007b403f64b44846e858b251cd21a reported cstate change: term changed from 0 to 1, leader changed from <none> to e26007b403f64b44846e858b251cd21a (127.0.98.1). New cstate: current_term: 1 leader_uuid: "e26007b403f64b44846e858b251cd21a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26007b403f64b44846e858b251cd21a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee51b18c18e4447ca61124e81521b7a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feef156dd078440cb1eb8fe78e845b8d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40753 } health_report { overall_health: UNKNOWN } } }
W20260927 12:41:45.239470 10966 tablet.cc:2369] T 37f035855cfb4d51a55be3f08e9b314e P ee51b18c18e4447ca61124e81521b7a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 12:41:45.251621 10832 tablet.cc:2369] T 37f035855cfb4d51a55be3f08e9b314e P e26007b403f64b44846e858b251cd21a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:41:45.291620 11111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:45.288394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46310 (local address 127.0.98.1:45083)
0927 12:41:45.288553 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 12:41:45.288556 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:41:45.288571 (+ 15us) server_negotiation.cc:407] Connection header received
0927 12:41:45.288651 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:41:45.288653 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:41:45.288704 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:41:45.288797 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:41:45.288975 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:45.289456 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:45.290250 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:41:45.290458 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:41:45.290566 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:41:45.290803 (+ 237us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:41:45.291215 (+ 412us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:41:45.291312 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 12:41:45.291358 (+ 46us) server_negotiation.cc:299] Negotiation successful
0927 12:41:45.291401 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 12:41:45.297502 10919 raft_consensus.cc:1261] T 37f035855cfb4d51a55be3f08e9b314e P ee51b18c18e4447ca61124e81521b7a6 [term 1 FOLLOWER]: Refusing update from remote peer e26007b403f64b44846e858b251cd21a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:41:45.297592 11053 raft_consensus.cc:1261] T 37f035855cfb4d51a55be3f08e9b314e P feef156dd078440cb1eb8fe78e845b8d [term 1 FOLLOWER]: Refusing update from remote peer e26007b403f64b44846e858b251cd21a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:41:45.298198 11123 consensus_queue.cc:1035] T 37f035855cfb4d51a55be3f08e9b314e P e26007b403f64b44846e858b251cd21a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee51b18c18e4447ca61124e81521b7a6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:41:45.298413 11129 consensus_queue.cc:1035] T 37f035855cfb4d51a55be3f08e9b314e P e26007b403f64b44846e858b251cd21a [LEADER]: Connected to new peer: Peer: permanent_uuid: "feef156dd078440cb1eb8fe78e845b8d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:41:45.305474 11132 mvcc.cc:205] Tried to move back new op lower bound from 7333940860091146240 to 7333940859792379904. Current Snapshot: MvccSnapshot[applied={T|T < 7333940860091146240}]
I20260927 12:41:45.305958 11133 mvcc.cc:205] Tried to move back new op lower bound from 7333940860091146240 to 7333940859792379904. Current Snapshot: MvccSnapshot[applied={T|T < 7333940860091146240}]
I20260927 12:41:45.314117 10638 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:41:45.314284 10638 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:41:45.316515 10638 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6971716913f14286bc4e04c193213b1b: Sending DeleteTablet for 3 replicas of tablet 37f035855cfb4d51a55be3f08e9b314e
I20260927 12:41:45.317198 10765 tablet_service.cc:1514] Processing DeleteTablet for tablet 37f035855cfb4d51a55be3f08e9b314e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:41:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46306
I20260927 12:41:45.317502 10899 tablet_service.cc:1514] Processing DeleteTablet for tablet 37f035855cfb4d51a55be3f08e9b314e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:41:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44476
I20260927 12:41:45.317914 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 10702
I20260927 12:41:45.317937 11142 tablet_replica.cc:330] T 37f035855cfb4d51a55be3f08e9b314e P ee51b18c18e4447ca61124e81521b7a6: stopping tablet replica
I20260927 12:41:45.317960 11033 tablet_service.cc:1514] Processing DeleteTablet for tablet 37f035855cfb4d51a55be3f08e9b314e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:41:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60594
I20260927 12:41:45.318729 11142 raft_consensus.cc:2229] T 37f035855cfb4d51a55be3f08e9b314e P ee51b18c18e4447ca61124e81521b7a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:41:45.319011 11142 raft_consensus.cc:2258] T 37f035855cfb4d51a55be3f08e9b314e P ee51b18c18e4447ca61124e81521b7a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:41:45.320255 11142 ts_tablet_manager.cc:1905] T 37f035855cfb4d51a55be3f08e9b314e P ee51b18c18e4447ca61124e81521b7a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:41:45.322315 11143 tablet_replica.cc:330] T 37f035855cfb4d51a55be3f08e9b314e P feef156dd078440cb1eb8fe78e845b8d: stopping tablet replica
I20260927 12:41:45.322646 11143 raft_consensus.cc:2229] T 37f035855cfb4d51a55be3f08e9b314e P feef156dd078440cb1eb8fe78e845b8d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:41:45.322975 11143 raft_consensus.cc:2258] T 37f035855cfb4d51a55be3f08e9b314e P feef156dd078440cb1eb8fe78e845b8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:41:45.323307 11142 ts_tablet_manager.cc:1918] T 37f035855cfb4d51a55be3f08e9b314e P ee51b18c18e4447ca61124e81521b7a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:41:45.323398 11142 log.cc:1196] T 37f035855cfb4d51a55be3f08e9b314e P ee51b18c18e4447ca61124e81521b7a6: Deleting WAL directory at /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/minicluster-data/ts-1/wal/wals/37f035855cfb4d51a55be3f08e9b314e
I20260927 12:41:45.323724 11142 ts_tablet_manager.cc:1939] T 37f035855cfb4d51a55be3f08e9b314e P ee51b18c18e4447ca61124e81521b7a6: Deleting consensus metadata
I20260927 12:41:45.324554 11143 ts_tablet_manager.cc:1905] T 37f035855cfb4d51a55be3f08e9b314e P feef156dd078440cb1eb8fe78e845b8d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:41:45.324909 10625 catalog_manager.cc:4841] TS ee51b18c18e4447ca61124e81521b7a6 (127.0.98.2:35467): tablet 37f035855cfb4d51a55be3f08e9b314e (table test-table [id=29420e005b224ddc9e43f35dc644686f]) successfully deleted
W20260927 12:41:45.325170 10625 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45083 (error 108)
I20260927 12:41:45.325459 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 10836
I20260927 12:41:45.327265 11143 ts_tablet_manager.cc:1918] T 37f035855cfb4d51a55be3f08e9b314e P feef156dd078440cb1eb8fe78e845b8d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:41:45.327342 11143 log.cc:1196] T 37f035855cfb4d51a55be3f08e9b314e P feef156dd078440cb1eb8fe78e845b8d: Deleting WAL directory at /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790512840836255-392-0/minicluster-data/ts-2/wal/wals/37f035855cfb4d51a55be3f08e9b314e
I20260927 12:41:45.327484 11112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:41:45.326001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45083 (local address 127.0.0.1:46314)
0927 12:41:45.327283 (+ 1282us) negotiation.cc:107] Waiting for socket to connect
0927 12:41:45.327385 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1200}
I20260927 12:41:45.327694 11143 ts_tablet_manager.cc:1939] T 37f035855cfb4d51a55be3f08e9b314e P feef156dd078440cb1eb8fe78e845b8d: Deleting consensus metadata
W20260927 12:41:45.327652 10625 catalog_manager.cc:4584] TS e26007b403f64b44846e858b251cd21a (127.0.98.1:45083): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 37f035855cfb4d51a55be3f08e9b314e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45083: connect: Connection refused (error 111)
I20260927 12:41:45.328727 10625 catalog_manager.cc:4841] TS feef156dd078440cb1eb8fe78e845b8d (127.0.98.3:40753): tablet 37f035855cfb4d51a55be3f08e9b314e (table test-table [id=29420e005b224ddc9e43f35dc644686f]) successfully deleted
I20260927 12:41:45.333629 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 10970
I20260927 12:41:45.339986 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 10608
2026-09-27T12:41:45Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2443 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:41:45 dist-test-slave-bnc1 krb5kdc[11147](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:41:45 dist-test-slave-bnc1 krb5kdc[11147](info): set up 2 sockets
Sep 27 12:41:45 dist-test-slave-bnc1 krb5kdc[11147](info): commencing operation
krb5kdc: starting...
W20260927 12:41:46.398680 392 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/krb5kdc/test-user.keytab.
Sep 27 12:41:46 dist-test-slave-bnc1 krb5kdc[11147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:41:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:41:46Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260927 12:41:46.563920 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task_Q2zcw/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/postgres -l logfile start
2026-09-27 12:41:49.103 UTC [11177] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 12:41:49.103 UTC [11177] LOG: listening on IPv4 address "127.0.98.20", port 43039
2026-09-27 12:41:49.107 UTC [11177] LOG: listening on Unix socket "/tmp/.s.PGSQL.43039"
2026-09-27 12:41:49.130 UTC [11180] LOG: database system was shut down at 2026-09-27 12:41:47 UTC
2026-09-27 12:41:49.133 UTC [11177] LOG: database system is ready to accept connections
I20260927 12:41:50.094259 392 mini_postgres.cc:96] Postgres bound to 43039
2026-09-27 12:41:50.100 UTC [11188] FATAL: database "slave" does not exist
127.0.98.20:43039 - accepting connections
I20260927 12:41:50.101048 392 mini_ranger.cc:161] Starting Ranger...
I20260927 12:41:50.115594 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 12:41:50.632367 392 mini_ranger.cc:87] Created ranger Postgres database
I20260927 12:41:50.632447 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/ranger-admin
2026-09-27 12:41:50,928 [I] DB FLAVOR :POSTGRES
2026-09-27 12:41:50,928 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 12:41:50,928 [I] Checking connection..
2026-09-27 12:41:50,928 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_Q2zcw/build/debug/bin/postgresql.jar:/tmp/dist-test-task_Q2zcw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43039/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 12:41:51,173 [I] Checking connection passed.
2026-09-27 12:41:51,173 [I] --------- Verifying version history table ---------
2026-09-27 12:41:51,173 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_Q2zcw/build/debug/bin/postgresql.jar:/tmp/dist-test-task_Q2zcw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43039/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 12:41:51,436 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 12:41:51,436 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_Q2zcw/build/debug/bin/postgresql.jar:/tmp/dist-test-task_Q2zcw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43039/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 12:41:51,699 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 12:41:51,699 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 12:41:51,699 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_Q2zcw/build/debug/bin/postgresql.jar:/tmp/dist-test-task_Q2zcw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43039/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task_Q2zcw/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 12:41:51.945 UTC [11252] WARNING: there is no transaction in progress
2026-09-27 12:41:51,951 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 12:41:51,952 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_Q2zcw/build/debug/bin/postgresql.jar:/tmp/dist-test-task_Q2zcw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43039/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 12:41:52,188 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 12:41:52,188 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 12:41:52,188 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_Q2zcw/build/debug/bin/postgresql.jar:/tmp/dist-test-task_Q2zcw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43039/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 12:41:52,426 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_Q2zcw/build/debug/bin/postgresql.jar:/tmp/dist-test-task_Q2zcw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43039/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 12:41:52,680 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_Q2zcw/build/debug/bin/postgresql.jar:/tmp/dist-test-task_Q2zcw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43039/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-bnc1.c.gcp-upstream.internal','N') ;"
2026-09-27 12:41:52,940 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_Q2zcw/build/debug/bin/postgresql.jar:/tmp/dist-test-task_Q2zcw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43039/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 12:41:53,186 [I] Table x_portal_user does not exist in database ranger
2026-09-27 12:41:53,187 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_Q2zcw/build/debug/bin/postgresql.jar:/tmp/dist-test-task_Q2zcw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43039/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 12:41:53,418 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 12:41:53,418 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_Q2zcw/build/debug/bin/postgresql.jar:/tmp/dist-test-task_Q2zcw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43039/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 12:41:53,646 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_Q2zcw/build/debug/bin/postgresql.jar:/tmp/dist-test-task_Q2zcw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43039/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 12:41:53,900 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 12:41:53,900 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_Q2zcw/build/debug/bin/postgresql.jar:/tmp/dist-test-task_Q2zcw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43039/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task_Q2zcw/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 12:41:54.902 UTC [11360] WARNING: there is no transaction in progress
2026-09-27 12:41:54.919 UTC [11360] WARNING: there is no transaction in progress
2026-09-27 12:41:54.937 UTC [11360] WARNING: there is no transaction in progress
2026-09-27 12:41:54.952 UTC [11360] WARNING: there is no transaction in progress
2026-09-27 12:41:54.970 UTC [11360] WARNING: there is no transaction in progress
2026-09-27 12:41:54.990 UTC [11360] WARNING: there is no transaction in progress
2026-09-27 12:41:55.045 UTC [11360] WARNING: there is no transaction in progress
2026-09-27 12:41:55.052 UTC [11360] WARNING: there is no transaction in progress
2026-09-27 12:41:55.067 UTC [11360] WARNING: there is no transaction in progress
2026-09-27 12:41:55.078 UTC [11360] WARNING: there is no transaction in progress
2026-09-27 12:41:55.089 UTC [11360] WARNING: there is no transaction in progress
2026-09-27 12:41:55.104 UTC [11360] WARNING: there is no transaction in progress
2026-09-27 12:41:55.114 UTC [11360] WARNING: there is no transaction in progress
2026-09-27 12:41:55.121 UTC [11360] WARNING: there is no transaction in progress
2026-09-27 12:41:55.129 UTC [11360] WARNING: there is no transaction in progress
2026-09-27 12:41:55.135 UTC [11360] WARNING: there is no transaction in progress
2026-09-27 12:41:55,878 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 12:41:55,878 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_Q2zcw/build/debug/bin/postgresql.jar:/tmp/dist-test-task_Q2zcw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43039/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 12:41:56,104 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 12:41:56,104 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_Q2zcw/build/debug/bin/postgresql.jar:/tmp/dist-test-task_Q2zcw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43039/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 12:41:56,362 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 12:41:56,362 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_Q2zcw/build/debug/bin/postgresql.jar:/tmp/dist-test-task_Q2zcw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43039/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 12:41:56,626 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 12:41:56,626 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_Q2zcw/build/debug/bin/postgresql.jar:/tmp/dist-test-task_Q2zcw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43039/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 12:41:56,883 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_Q2zcw/build/debug/bin/postgresql.jar:/tmp/dist-test-task_Q2zcw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43039/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 12:41:57,117 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_Q2zcw/build/debug/bin/postgresql.jar:/tmp/dist-test-task_Q2zcw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43039/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-bnc1.c.gcp-upstream.internal';"
2026-09-27 12:41:57,361 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 12:41:57,361 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_Q2zcw/build/debug/bin/postgresql.jar:/tmp/dist-test-task_Q2zcw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43039/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 12:41:57,596 [I] DB_PATCHES have already been applied
I20260927 12:41:57.602350 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/ranger-admin:/tmp/dist-test-task_Q2zcw/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task_Q2zcw/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task_Q2zcw/build/debug/bin/postgresql.jar:/tmp/dist-test-task_Q2zcw/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 12:41:57.602415 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260927 12:41:57.605980 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:60711
Sep 27, 2026 12:41:57 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 12:41:57 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 12:41:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 12:41:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 12:41:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 12:41:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 12:41:58 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 12:41:58.716054 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.615s user 0.000s sys 0.010s
Sep 27, 2026 12:42:18 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 12:42:18 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 27, 2026 12:42:18 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 12:42:18 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 12:42:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 12:42:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 12:42:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 12:42:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 12:42:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 12:42:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 12:42:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 12:42:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 12:42:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 12:42:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 12:42:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 12:42:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 12:42:18 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 12:42:19 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 12:42:20.036933 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 12:42:20 dist-test-slave-bnc1 krb5kdc[11147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512940, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 12:42:20.125699 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 12:42:20.135855 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 12:42:20.964680 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 12:42:20.964779 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/ranger-kms
I20260927 12:42:20.964896 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:43039
2026-09-27 12:42:21,169 [I] DB FLAVOR :POSTGRES
2026-09-27 12:42:21,169 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 12:42:21,170 [I] Checking connection
2026-09-27 12:42:21,170 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_Q2zcw/build/debug/bin/postgresql.jar:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43039/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 12:42:21,407 [I] connection success
2026-09-27 12:42:21,407 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 12:42:21,407 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 12:42:21,408 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_Q2zcw/build/debug/bin/postgresql.jar:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43039/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 12:42:21,655 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 12:42:21,656 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 12:42:21,656 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 12:42:21,656 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_Q2zcw/build/debug/bin/postgresql.jar:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43039/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 12:42:21,910 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 12:42:22.396436 392 mini_ranger_kms.cc:310] Created kms service
I20260927 12:42:22.595367 392 mini_ranger_kms.cc:326] Created kudu user
I20260927 12:42:22.642037 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 12:42:22 dist-test-slave-bnc1 krb5kdc[11147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512940, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 12:42:23.038394 392 mini_ranger_kms.cc:384] Added ranger policy
I20260927 12:42:23.038545 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/ranger-kms:/tmp/dist-test-task_Q2zcw/build/debug/bin/postgresql.jar:/tmp/dist-test-task_Q2zcw/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task_Q2zcw/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 12:42:23.038568 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260927 12:42:23.042577 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11540
I20260927 12:42:23.042693 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:41945
Sep 27, 2026 12:42:24 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 12:42:24 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 12:42:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 12:42:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 12:42:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 12:42:24 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 12:42:25.337973 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.212s user 0.012s sys 0.172s
I20260927 12:42:25.338117 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 12:42:25.338244 392 mini_ranger_kms.cc:401] 127.0.98.20:41945/kms/v1/keys
Sep 27 12:42:25 dist-test-slave-bnc1 krb5kdc[11147](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790512945, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 12:42:25 dist-test-slave-bnc1 krb5kdc[11147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512945, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 12:42:26 dist-test-slave-bnc1 krb5kdc[11147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512945, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 12:42:28 dist-test-slave-bnc1 krb5kdc[11147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:42:28.560530 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46199
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46199
--ranger_config_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:41945/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:42:28.678947 11591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:42:28.679201 11591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:42:28.679276 11591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:42:28.683248 11591 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:42:28.683297 11591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:42:28.683342 11591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:42:28.683373 11591 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:42:28.683408 11591 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:42:28.688710 11591 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:41945/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46199
--ranger_config_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11591
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:42:28.689769 11591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:42:28.690747 11591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:42:28.698952 11597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:42:28.700414 11591 server_base.cc:1034] running on GCE node
W20260927 12:42:28.700647 11599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:42:28.701036 11596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:42:28.701610 11591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:42:28.702667 11591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:42:28.703848 11591 hybrid_clock.cc:648] HybridClock initialized: now 1790512948703834 us; error 30 us; skew 500 ppm
Sep 27 12:42:28 dist-test-slave-bnc1 krb5kdc[11147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512948, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:42:28.709998 11591 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:42:28.710880 11591 webserver.cc:460] Webserver started at http://127.0.98.62:36861/ using document root <none> and password file <none>
I20260927 12:42:28.711410 11591 fs_manager.cc:359] Metadata directory not provided
I20260927 12:42:28.711488 11591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:42:28.711705 11591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 12:42:28 dist-test-slave-bnc1 krb5kdc[11147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512948, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 12:42:28.759433 11591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/master-0/data/instance:
uuid: "6a85f0a52e104948a1c5748817fd2878"
format_stamp: "Formatted at 2026-09-27 12:42:28 on dist-test-slave-bnc1"
server_key: "0129665af8d1947e19037a4b7c311dfe"
server_key_iv: "af0d718361a10bcc1000ed8672967561"
server_key_version: "kuduclusterkey@0"
I20260927 12:42:28.760443 11591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/master-0/wal/instance:
uuid: "6a85f0a52e104948a1c5748817fd2878"
format_stamp: "Formatted at 2026-09-27 12:42:28 on dist-test-slave-bnc1"
server_key: "0129665af8d1947e19037a4b7c311dfe"
server_key_iv: "af0d718361a10bcc1000ed8672967561"
server_key_version: "kuduclusterkey@0"
I20260927 12:42:28.766343 11591 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 12:42:28.781618 11606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:42:28.783032 11591 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:42:28.783169 11591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/master-0/wal
uuid: "6a85f0a52e104948a1c5748817fd2878"
format_stamp: "Formatted at 2026-09-27 12:42:28 on dist-test-slave-bnc1"
server_key: "0129665af8d1947e19037a4b7c311dfe"
server_key_iv: "af0d718361a10bcc1000ed8672967561"
server_key_version: "kuduclusterkey@0"
I20260927 12:42:28.783305 11591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:42:28.800854 11591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:42:28.805335 11591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:42:28.805549 11591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:42:28.814999 11591 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46199
I20260927 12:42:28.815001 11658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46199 every 8 connection(s)
I20260927 12:42:28.816053 11591 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:42:28.819588 11659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:42:28.819978 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 11591
I20260927 12:42:28.820111 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/master-0/wal/instance
I20260927 12:42:28.820437 392 external_mini_cluster.cc:1428] Setting key 2b034c70d2fbbe5433295061561b37d4
I20260927 12:42:28.828116 11659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a85f0a52e104948a1c5748817fd2878: Bootstrap starting.
I20260927 12:42:28.830991 11659 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a85f0a52e104948a1c5748817fd2878: Neither blocks nor log segments found. Creating new log.
Sep 27 12:42:28 dist-test-slave-bnc1 krb5kdc[11147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:42:28.832078 11659 log.cc:824] T 00000000000000000000000000000000 P 6a85f0a52e104948a1c5748817fd2878: Log is configured to *not* fsync() on all Append() calls
I20260927 12:42:28.834359 11659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a85f0a52e104948a1c5748817fd2878: No bootstrap required, opened a new log
I20260927 12:42:28.838006 11659 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a85f0a52e104948a1c5748817fd2878 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a85f0a52e104948a1c5748817fd2878" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46199 } }
I20260927 12:42:28.838281 11659 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a85f0a52e104948a1c5748817fd2878 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:42:28.838352 11659 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a85f0a52e104948a1c5748817fd2878 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a85f0a52e104948a1c5748817fd2878, State: Initialized, Role: FOLLOWER
I20260927 12:42:28.838653 11662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:42:28.821933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42230 (local address 127.0.98.62:46199)
0927 12:42:28.822372 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0927 12:42:28.822387 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 12:42:28.824427 (+ 2040us) server_negotiation.cc:407] Connection header received
0927 12:42:28.825220 (+ 793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:42:28.825238 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:42:28.825642 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:42:28.825991 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:42:28.826773 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:42:28.827744 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:42:28.828706 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:42:28.829195 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:42:28.829368 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:42:28.832523 (+ 3155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:42:28.832528 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:42:28.832542 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:42:28.832569 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:42:28.835268 (+ 2699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:42:28.835802 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:42:28.835808 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:42:28.835815 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:42:28.835951 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:42:28.836266 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:42:28.836270 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:42:28.836271 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:42:28.836820 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:42:28.837014 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0927 12:42:28.837325 (+ 311us) server_negotiation.cc:299] Negotiation successful
0927 12:42:28.837657 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":150,"threads_started":1}
I20260927 12:42:28.838891 11659 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a85f0a52e104948a1c5748817fd2878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a85f0a52e104948a1c5748817fd2878" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46199 } }
I20260927 12:42:28.839077 11659 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a85f0a52e104948a1c5748817fd2878 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:42:28.839151 11659 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a85f0a52e104948a1c5748817fd2878 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:42:28.839255 11659 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a85f0a52e104948a1c5748817fd2878 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:42:28.840371 11659 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a85f0a52e104948a1c5748817fd2878 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a85f0a52e104948a1c5748817fd2878" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46199 } }
I20260927 12:42:28.840711 11659 leader_election.cc:302] T 00000000000000000000000000000000 P 6a85f0a52e104948a1c5748817fd2878 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a85f0a52e104948a1c5748817fd2878; no voters:
I20260927 12:42:28.841066 11659 leader_election.cc:288] T 00000000000000000000000000000000 P 6a85f0a52e104948a1c5748817fd2878 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:42:28.841439 11665 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a85f0a52e104948a1c5748817fd2878 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:42:28.841706 11665 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a85f0a52e104948a1c5748817fd2878 [term 1 LEADER]: Becoming Leader. State: Replica: 6a85f0a52e104948a1c5748817fd2878, State: Running, Role: LEADER
I20260927 12:42:28.842077 11665 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a85f0a52e104948a1c5748817fd2878 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a85f0a52e104948a1c5748817fd2878" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46199 } }
I20260927 12:42:28.842335 11659 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a85f0a52e104948a1c5748817fd2878 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:42:28.844033 11666 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a85f0a52e104948a1c5748817fd2878 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a85f0a52e104948a1c5748817fd2878" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a85f0a52e104948a1c5748817fd2878" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46199 } } }
I20260927 12:42:28.844148 11666 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a85f0a52e104948a1c5748817fd2878 [sys.catalog]: This master's current role is: LEADER
I20260927 12:42:28.844338 11666 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a85f0a52e104948a1c5748817fd2878 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a85f0a52e104948a1c5748817fd2878. Latest consensus state: current_term: 1 leader_uuid: "6a85f0a52e104948a1c5748817fd2878" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a85f0a52e104948a1c5748817fd2878" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46199 } } }
I20260927 12:42:28.844404 11666 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a85f0a52e104948a1c5748817fd2878 [sys.catalog]: This master's current role is: LEADER
I20260927 12:42:28.844548 11659 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 12:42:28.845360 11678 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:42:28.851270 11678 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:42:28.856943 11678 catalog_manager.cc:1347] Generated new cluster ID: a3957cc13f3e4d68b85a9d4e797ccacb
I20260927 12:42:28.857057 11678 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:42:28.870878 11678 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:42:28.872224 11678 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:42:28.879554 11678 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a85f0a52e104948a1c5748817fd2878: Generated new TSK 0
I20260927 12:42:28.880374 11678 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 12:42:29.540544 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:41945/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46199
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:42:29.681377 11704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:42:29.681758 11704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:42:29.681829 11704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:42:29.687448 11704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:42:29.687536 11704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:42:29.687636 11704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:42:29.694298 11704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:41945/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:42:29.695647 11704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:42:29.696995 11704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:42:29.709275 11711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:42:29.709340 11712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:42:29.709478 11704 server_base.cc:1034] running on GCE node
W20260927 12:42:29.709507 11714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:42:29.710023 11704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:42:29.710700 11704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:42:29.711874 11704 hybrid_clock.cc:648] HybridClock initialized: now 1790512949711849 us; error 38 us; skew 500 ppm
Sep 27 12:42:29 dist-test-slave-bnc1 krb5kdc[11147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512949, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:42:29.714848 11704 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 12:42:29.715869 11704 webserver.cc:460] Webserver started at http://127.0.98.1:42109/ using document root <none> and password file <none>
I20260927 12:42:29.716491 11704 fs_manager.cc:359] Metadata directory not provided
I20260927 12:42:29.716575 11704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:42:29.716814 11704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 12:42:29 dist-test-slave-bnc1 krb5kdc[11147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512949, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 12:42:29.735417 11704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/ts-0/data/instance:
uuid: "dc74971c016545978123de69450d214f"
format_stamp: "Formatted at 2026-09-27 12:42:29 on dist-test-slave-bnc1"
server_key: "92c442dafeaa8c8284785aa5f09a823b"
server_key_iv: "779c7f2469fd2f03f81eac73ba45d6f8"
server_key_version: "kuduclusterkey@0"
I20260927 12:42:29.736227 11704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/ts-0/wal/instance:
uuid: "dc74971c016545978123de69450d214f"
format_stamp: "Formatted at 2026-09-27 12:42:29 on dist-test-slave-bnc1"
server_key: "92c442dafeaa8c8284785aa5f09a823b"
server_key_iv: "779c7f2469fd2f03f81eac73ba45d6f8"
server_key_version: "kuduclusterkey@0"
I20260927 12:42:29.741448 11704 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260927 12:42:29.765825 11722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:42:29.771306 11704 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.002s sys 0.001s
I20260927 12:42:29.771466 11704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/ts-0/wal
uuid: "dc74971c016545978123de69450d214f"
format_stamp: "Formatted at 2026-09-27 12:42:29 on dist-test-slave-bnc1"
server_key: "92c442dafeaa8c8284785aa5f09a823b"
server_key_iv: "779c7f2469fd2f03f81eac73ba45d6f8"
server_key_version: "kuduclusterkey@0"
I20260927 12:42:29.771616 11704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:42:29.800836 11704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:42:29.810747 11704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:42:29.811009 11704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:42:29.811734 11704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:42:29.812788 11704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:42:29.812889 11704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:42:29.813014 11704 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:42:29.813097 11704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:42:29.823568 11704 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39385
I20260927 12:42:29.827625 11704 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:42:29.830793 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 11704
I20260927 12:42:29.830910 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/ts-0/wal/instance
I20260927 12:42:29.832489 392 external_mini_cluster.cc:1428] Setting key b8ee68f0d480a6a8ae52708fdab0a811
I20260927 12:42:29.829501 11836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39385 every 8 connection(s)
Sep 27 12:42:29 dist-test-slave-bnc1 krb5kdc[11147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512949, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 12:42:29.865213 11841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:42:29.838460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44115 (local address 127.0.98.62:46199)
0927 12:42:29.841717 (+ 3257us) server_negotiation.cc:206] Beginning negotiation
0927 12:42:29.841725 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 12:42:29.842217 (+ 492us) server_negotiation.cc:407] Connection header received
0927 12:42:29.843333 (+ 1116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:42:29.843336 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:42:29.843424 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:42:29.843510 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:42:29.850393 (+ 6883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:42:29.851290 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:42:29.853037 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:42:29.853410 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:42:29.853492 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:42:29.858317 (+ 4825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:42:29.858322 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:42:29.858325 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:42:29.858365 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:42:29.860689 (+ 2324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:42:29.862199 (+ 1510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:42:29.862204 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:42:29.862206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:42:29.862389 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:42:29.863386 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:42:29.863391 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:42:29.863393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:42:29.863681 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:42:29.863785 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 12:42:29.864813 (+ 1028us) server_negotiation.cc:299] Negotiation successful
0927 12:42:29.864978 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3186,"thread_start_us":82,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 12:42:29.870245 11840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:42:29.837352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46199 (local address 127.0.98.1:44115)
0927 12:42:29.841801 (+ 4449us) negotiation.cc:107] Waiting for socket to connect
0927 12:42:29.841857 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 12:42:29.842965 (+ 1108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:42:29.843844 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:42:29.843853 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:42:29.844227 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:42:29.850191 (+ 5964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:42:29.850216 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:42:29.851445 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:42:29.851449 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:42:29.852894 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:42:29.852905 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:42:29.853734 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:42:29.853738 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:42:29.853942 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:42:29.854651 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:42:29.854673 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:42:29.858037 (+ 3364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:42:29.860879 (+ 2842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:42:29.860887 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:42:29.860907 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:42:29.861407 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:42:29.862547 (+ 1140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:42:29.862551 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:42:29.862553 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:42:29.862773 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:42:29.863867 (+ 1094us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:42:29.863875 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:42:29.864278 (+ 403us) client_negotiation.cc:769] Sending connection context
0927 12:42:29.865896 (+ 1618us) client_negotiation.cc:240] Negotiation successful
0927 12:42:29.869042 (+ 3146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4025,"thread_start_us":165,"threads_started":1}
I20260927 12:42:29.872088 11837 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46199
I20260927 12:42:29.872336 11837 heartbeater.cc:464] Registering TS with master...
I20260927 12:42:29.872920 11837 heartbeater.cc:511] Master 127.0.98.62:46199 requested a full tablet report, sending...
I20260927 12:42:29.874714 11617 ts_manager.cc:194] Registered new tserver with Master: dc74971c016545978123de69450d214f (127.0.98.1:39385)
I20260927 12:42:29.876271 11617 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44115
Sep 27 12:42:29 dist-test-slave-bnc1 krb5kdc[11147](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790512949, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 12:42:29.906036 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:41945/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46199
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:42:30.072574 11843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:42:30.073031 11843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:42:30.073124 11843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:42:30.080231 11843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:42:30.080317 11843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:42:30.080412 11843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:42:30.089583 11843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:41945/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11843
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:42:30.091125 11843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:42:30.092465 11843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:42:30.105674 11850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:42:30.106057 11849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:42:30.106421 11852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:42:30.106758 11843 server_base.cc:1034] running on GCE node
I20260927 12:42:30.107231 11843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:42:30.108019 11843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:42:30.109174 11843 hybrid_clock.cc:648] HybridClock initialized: now 1790512950109104 us; error 86 us; skew 500 ppm
Sep 27 12:42:30 dist-test-slave-bnc1 krb5kdc[11147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512950, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:42:30.112890 11843 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 12:42:30.114135 11843 webserver.cc:460] Webserver started at http://127.0.98.2:39663/ using document root <none> and password file <none>
I20260927 12:42:30.114898 11843 fs_manager.cc:359] Metadata directory not provided
I20260927 12:42:30.114995 11843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:42:30.115321 11843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 12:42:30 dist-test-slave-bnc1 krb5kdc[11147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512950, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 12:42:30.137192 11843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/ts-1/data/instance:
uuid: "f399a82a7f0c4f40950da3bf58a6279d"
format_stamp: "Formatted at 2026-09-27 12:42:30 on dist-test-slave-bnc1"
server_key: "85dc6960ff508d0fcd4be7165d93a0fc"
server_key_iv: "d6fc139d6a2ecc1bf65b04bd8fce9391"
server_key_version: "kuduclusterkey@0"
I20260927 12:42:30.138048 11843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f399a82a7f0c4f40950da3bf58a6279d"
format_stamp: "Formatted at 2026-09-27 12:42:30 on dist-test-slave-bnc1"
server_key: "85dc6960ff508d0fcd4be7165d93a0fc"
server_key_iv: "d6fc139d6a2ecc1bf65b04bd8fce9391"
server_key_version: "kuduclusterkey@0"
I20260927 12:42:30.143857 11843 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 12:42:30.161844 11859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:42:30.163980 11843 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 12:42:30.164137 11843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/ts-1/wal
uuid: "f399a82a7f0c4f40950da3bf58a6279d"
format_stamp: "Formatted at 2026-09-27 12:42:30 on dist-test-slave-bnc1"
server_key: "85dc6960ff508d0fcd4be7165d93a0fc"
server_key_iv: "d6fc139d6a2ecc1bf65b04bd8fce9391"
server_key_version: "kuduclusterkey@0"
I20260927 12:42:30.164408 11843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:42:30.193491 11843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:42:30.197906 11843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:42:30.198190 11843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:42:30.199038 11843 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:42:30.200307 11843 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:42:30.200417 11843 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:42:30.200522 11843 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:42:30.200613 11843 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:42:30.212178 11843 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38131
I20260927 12:42:30.213413 11843 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:42:30.215328 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 11843
I20260927 12:42:30.215459 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/ts-1/wal/instance
I20260927 12:42:30.215811 392 external_mini_cluster.cc:1428] Setting key aff6434ad57aa725e761cd3c77b98ad6
I20260927 12:42:30.225898 11972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38131 every 8 connection(s)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Sep 27 12:42:30 dist-test-slave-bnc1 krb5kdc[11147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512950, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:42:30.248605 11976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:42:30.229515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46199 (local address 127.0.98.2:48039)
0927 12:42:30.231773 (+ 2258us) negotiation.cc:107] Waiting for socket to connect
0927 12:42:30.231852 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0927 12:42:30.233086 (+ 1234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:42:30.233998 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:42:30.234009 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:42:30.234391 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:42:30.235332 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:42:30.235352 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:42:30.236637 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:42:30.236640 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:42:30.238392 (+ 1752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:42:30.238403 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:42:30.239294 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:42:30.239299 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:42:30.239637 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:42:30.240408 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:42:30.240431 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:42:30.242454 (+ 2023us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:42:30.244851 (+ 2397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:42:30.244858 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:42:30.244878 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:42:30.245256 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:42:30.245659 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:42:30.245663 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:42:30.245665 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:42:30.245864 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:42:30.246396 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:42:30.246403 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:42:30.246792 (+ 389us) client_negotiation.cc:769] Sending connection context
0927 12:42:30.247016 (+ 224us) client_negotiation.cc:240] Negotiation successful
0927 12:42:30.247392 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1596,"thread_start_us":201,"threads_started":1}
I20260927 12:42:30.249261 11841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:42:30.229778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48039 (local address 127.0.98.62:46199)
0927 12:42:30.229926 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 12:42:30.229931 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:42:30.231982 (+ 2051us) server_negotiation.cc:407] Connection header received
0927 12:42:30.233378 (+ 1396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:42:30.233382 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:42:30.233443 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:42:30.233502 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:42:30.235518 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:42:30.236447 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:42:30.238564 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:42:30.238970 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:42:30.239065 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:42:30.242669 (+ 3604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:42:30.242673 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:42:30.242676 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:42:30.242711 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:42:30.244700 (+ 1989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:42:30.245387 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:42:30.245391 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:42:30.245394 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:42:30.245489 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:42:30.245985 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:42:30.245989 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:42:30.245991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:42:30.246236 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:42:30.246339 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 12:42:30.248932 (+ 2593us) server_negotiation.cc:299] Negotiation successful
0927 12:42:30.249077 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 12:42:30.251276 11973 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46199
I20260927 12:42:30.251566 11973 heartbeater.cc:464] Registering TS with master...
I20260927 12:42:30.252216 11973 heartbeater.cc:511] Master 127.0.98.62:46199 requested a full tablet report, sending...
I20260927 12:42:30.253590 11617 ts_manager.cc:194] Registered new tserver with Master: f399a82a7f0c4f40950da3bf58a6279d (127.0.98.2:38131)
I20260927 12:42:30.254405 11617 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48039
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 12:42:30.274456 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:41945/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46199
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 12:42:30 dist-test-slave-bnc1 krb5kdc[11147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512949, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260927 12:42:30.448475 11978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:42:30.448853 11978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:42:30.448945 11978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:42:30.454924 11978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:42:30.455000 11978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:42:30.455120 11978 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:42:30.462743 11978 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:41945/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:42:30.464074 11978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:42:30.465397 11978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:42:30.478768 11983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:42:30.479326 11978 server_base.cc:1034] running on GCE node
W20260927 12:42:30.479486 11984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:42:30.481958 11986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:42:30.482468 11978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:42:30.483280 11978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:42:30.499488 11978 hybrid_clock.cc:648] HybridClock initialized: now 1790512950498684 us; error 824 us; skew 500 ppm
Sep 27 12:42:30 dist-test-slave-bnc1 krb5kdc[11147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512950, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:42:30.503576 11978 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 12:42:30.504727 11978 webserver.cc:460] Webserver started at http://127.0.98.3:42157/ using document root <none> and password file <none>
I20260927 12:42:30.505486 11978 fs_manager.cc:359] Metadata directory not provided
I20260927 12:42:30.505604 11978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:42:30.505864 11978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 12:42:30 dist-test-slave-bnc1 krb5kdc[11147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512950, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 12:42:30.524317 11978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/ts-2/data/instance:
uuid: "95604c169d774c85858abe84ec8d3d28"
format_stamp: "Formatted at 2026-09-27 12:42:30 on dist-test-slave-bnc1"
server_key: "f4d4319c0f10a00f3da23df4cb415076"
server_key_iv: "e4488ae0631620562187efd035ccf15c"
server_key_version: "kuduclusterkey@0"
I20260927 12:42:30.525143 11978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/ts-2/wal/instance:
uuid: "95604c169d774c85858abe84ec8d3d28"
format_stamp: "Formatted at 2026-09-27 12:42:30 on dist-test-slave-bnc1"
server_key: "f4d4319c0f10a00f3da23df4cb415076"
server_key_iv: "e4488ae0631620562187efd035ccf15c"
server_key_version: "kuduclusterkey@0"
I20260927 12:42:30.539283 11978 fs_manager.cc:691] Time spent creating directory manager: real 0.014s user 0.004s sys 0.000s
I20260927 12:42:30.557739 11993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:42:30.559386 11978 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:42:30.559508 11978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/ts-2/wal
uuid: "95604c169d774c85858abe84ec8d3d28"
format_stamp: "Formatted at 2026-09-27 12:42:30 on dist-test-slave-bnc1"
server_key: "f4d4319c0f10a00f3da23df4cb415076"
server_key_iv: "e4488ae0631620562187efd035ccf15c"
server_key_version: "kuduclusterkey@0"
I20260927 12:42:30.559648 11978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:42:30.605106 11978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:42:30.625520 11978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:42:30.625866 11978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:42:30.626636 11978 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:42:30.627724 11978 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:42:30.627830 11978 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:42:30.627926 11978 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:42:30.627990 11978 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:42:30.643823 11978 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33735
I20260927 12:42:30.645113 11978 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:42:30.648484 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 11978
I20260927 12:42:30.648614 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/ts-2/wal/instance
I20260927 12:42:30.648900 392 external_mini_cluster.cc:1428] Setting key defe1bb6253a8a25178817dee16b7a5c
I20260927 12:42:30.653362 12106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33735 every 8 connection(s)
Sep 27 12:42:30 dist-test-slave-bnc1 krb5kdc[11147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512950, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:42:30.678788 11841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:42:30.659127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39275 (local address 127.0.98.62:46199)
0927 12:42:30.659250 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 12:42:30.659255 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:42:30.663519 (+ 4264us) server_negotiation.cc:407] Connection header received
0927 12:42:30.664883 (+ 1364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:42:30.664887 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:42:30.664939 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:42:30.664990 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:42:30.666888 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:42:30.667746 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:42:30.669473 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:42:30.669854 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:42:30.669947 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:42:30.673404 (+ 3457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:42:30.673408 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:42:30.673411 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:42:30.673438 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:42:30.676373 (+ 2935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:42:30.676993 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:42:30.676997 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:42:30.676999 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:42:30.677082 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:42:30.677476 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:42:30.677479 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:42:30.677481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:42:30.677721 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:42:30.677814 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 12:42:30.678488 (+ 674us) server_negotiation.cc:299] Negotiation successful
0927 12:42:30.678625 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 12:42:30.680630 12109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:42:30.657979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46199 (local address 127.0.98.3:39275)
0927 12:42:30.663368 (+ 5389us) negotiation.cc:107] Waiting for socket to connect
0927 12:42:30.663429 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0927 12:42:30.664544 (+ 1115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:42:30.665460 (+ 916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:42:30.665470 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:42:30.665974 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:42:30.666722 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:42:30.666742 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:42:30.667905 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:42:30.667909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:42:30.669329 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:42:30.669339 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:42:30.670167 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:42:30.670170 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:42:30.670629 (+ 459us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:42:30.671410 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:42:30.671431 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:42:30.673212 (+ 1781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:42:30.676518 (+ 3306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:42:30.676525 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:42:30.676540 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:42:30.676871 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:42:30.677195 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:42:30.677198 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:42:30.677201 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:42:30.677367 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:42:30.677900 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:42:30.677907 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:42:30.678203 (+ 296us) client_negotiation.cc:769] Sending connection context
0927 12:42:30.679229 (+ 1026us) client_negotiation.cc:240] Negotiation successful
0927 12:42:30.679552 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3717,"thread_start_us":167,"threads_started":1}
I20260927 12:42:30.682125 12107 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46199
I20260927 12:42:30.682372 12107 heartbeater.cc:464] Registering TS with master...
I20260927 12:42:30.682971 12107 heartbeater.cc:511] Master 127.0.98.62:46199 requested a full tablet report, sending...
I20260927 12:42:30.684146 11617 ts_manager.cc:194] Registered new tserver with Master: 95604c169d774c85858abe84ec8d3d28 (127.0.98.3:33735)
I20260927 12:42:30.684924 11617 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39275
I20260927 12:42:30.688481 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 12:42:30.700270 11841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:42:30.690848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42246 (local address 127.0.98.62:46199)
0927 12:42:30.690983 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 12:42:30.690988 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:42:30.691433 (+ 445us) server_negotiation.cc:407] Connection header received
0927 12:42:30.691555 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:42:30.691570 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:42:30.691619 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:42:30.691707 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:42:30.692533 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:42:30.693399 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:42:30.694655 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:42:30.694989 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:42:30.695102 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:42:30.696735 (+ 1633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:42:30.696739 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:42:30.696742 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:42:30.696767 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:42:30.698557 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:42:30.699024 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:42:30.699027 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:42:30.699029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:42:30.699116 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:42:30.699407 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:42:30.699410 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:42:30.699412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:42:30.699624 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:42:30.699757 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 12:42:30.699987 (+ 230us) server_negotiation.cc:299] Negotiation successful
0927 12:42:30.700105 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 12:42:30.703464 11617 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42246:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 12:42:30.708101 11617 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:42:30.740247 12121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:42:30.722610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38131 (local address 127.0.0.1:34600)
0927 12:42:30.725335 (+ 2725us) negotiation.cc:107] Waiting for socket to connect
0927 12:42:30.725347 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 12:42:30.725421 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:42:30.728253 (+ 2832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:42:30.728256 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:42:30.728273 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:42:30.728362 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:42:30.728368 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:42:30.731423 (+ 3055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:42:30.731426 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:42:30.736715 (+ 5289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:42:30.736721 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:42:30.739664 (+ 2943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:42:30.739668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:42:30.739790 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:42:30.739832 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 12:42:30.739951 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 12:42:30.740079 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2666,"thread_start_us":85,"threads_started":1}
I20260927 12:42:30.741900 12119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:42:30.721029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33735 (local address 127.0.0.1:49884)
0927 12:42:30.724887 (+ 3858us) negotiation.cc:107] Waiting for socket to connect
0927 12:42:30.724943 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 12:42:30.725073 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:42:30.727194 (+ 2121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:42:30.727201 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:42:30.727224 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:42:30.727357 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:42:30.727363 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:42:30.734788 (+ 7425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:42:30.734791 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:42:30.736121 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:42:30.736132 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:42:30.741457 (+ 5325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:42:30.741461 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:42:30.741558 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:42:30.741610 (+ 52us) client_negotiation.cc:769] Sending connection context
0927 12:42:30.741682 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 12:42:30.741755 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3669,"thread_start_us":111,"threads_started":1}
I20260927 12:42:30.742650 12120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:42:30.721880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49884 (local address 127.0.98.3:33735)
0927 12:42:30.726499 (+ 4619us) server_negotiation.cc:206] Beginning negotiation
0927 12:42:30.726514 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 12:42:30.726542 (+ 28us) server_negotiation.cc:407] Connection header received
0927 12:42:30.726633 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:42:30.726639 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:42:30.726864 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:42:30.727019 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:42:30.728496 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:42:30.734638 (+ 6142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:42:30.736860 (+ 2222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:42:30.741141 (+ 4281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:42:30.741252 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:42:30.741395 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 12:42:30.742378 (+ 983us) server_negotiation.cc:299] Negotiation successful
0927 12:42:30.742493 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3730,"thread_start_us":96,"threads_started":1}
I20260927 12:42:30.747586 12041 tablet_service.cc:1467] Processing CreateTablet for tablet 046d11cf6e84431b8ff97750193569df (DEFAULT_TABLE table=test-table [id=94f1ccb7c0af4e63b885fc72cb2effc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:42:30.748893 12041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 046d11cf6e84431b8ff97750193569df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:42:30.757476 12122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:42:30.722990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34600 (local address 127.0.98.2:38131)
0927 12:42:30.727647 (+ 4657us) server_negotiation.cc:206] Beginning negotiation
0927 12:42:30.727665 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 12:42:30.727688 (+ 23us) server_negotiation.cc:407] Connection header received
0927 12:42:30.727759 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:42:30.727765 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:42:30.727964 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:42:30.728110 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:42:30.729861 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:42:30.731275 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:42:30.738382 (+ 7107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:42:30.739313 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:42:30.739426 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:42:30.739601 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0927 12:42:30.742901 (+ 3300us) server_negotiation.cc:299] Negotiation successful
0927 12:42:30.743016 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4530,"thread_start_us":61,"threads_started":1}
I20260927 12:42:30.761148 11907 tablet_service.cc:1467] Processing CreateTablet for tablet 046d11cf6e84431b8ff97750193569df (DEFAULT_TABLE table=test-table [id=94f1ccb7c0af4e63b885fc72cb2effc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:42:30.762487 11907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 046d11cf6e84431b8ff97750193569df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:42:30.768486 12123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:42:30.724141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39385 (local address 127.0.0.1:52568)
0927 12:42:30.736378 (+ 12237us) negotiation.cc:107] Waiting for socket to connect
0927 12:42:30.736392 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:42:30.736470 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:42:30.760477 (+ 24007us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:42:30.760481 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:42:30.760502 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:42:30.760636 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:42:30.760643 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:42:30.765186 (+ 4543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:42:30.765190 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:42:30.766759 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:42:30.766773 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:42:30.768026 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:42:30.768030 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:42:30.768143 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:42:30.768175 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 12:42:30.768234 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 12:42:30.768315 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12162,"thread_start_us":94,"threads_started":1}
I20260927 12:42:30.770262 12124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:42:30.724537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52568 (local address 127.0.98.1:39385)
0927 12:42:30.759962 (+ 35425us) server_negotiation.cc:206] Beginning negotiation
0927 12:42:30.759969 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 12:42:30.759998 (+ 29us) server_negotiation.cc:407] Connection header received
0927 12:42:30.760083 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:42:30.760088 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:42:30.760303 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:42:30.760806 (+ 503us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:42:30.760944 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:42:30.763291 (+ 2347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:42:30.766934 (+ 3643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:42:30.767797 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:42:30.768831 (+ 1034us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:42:30.768982 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 12:42:30.769035 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 12:42:30.770116 (+ 1081us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35302,"thread_start_us":73,"threads_started":1}
I20260927 12:42:30.770982 12125 tablet_bootstrap.cc:492] T 046d11cf6e84431b8ff97750193569df P 95604c169d774c85858abe84ec8d3d28: Bootstrap starting.
I20260927 12:42:30.773351 11770 tablet_service.cc:1467] Processing CreateTablet for tablet 046d11cf6e84431b8ff97750193569df (DEFAULT_TABLE table=test-table [id=94f1ccb7c0af4e63b885fc72cb2effc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:42:30.773964 12125 tablet_bootstrap.cc:654] T 046d11cf6e84431b8ff97750193569df P 95604c169d774c85858abe84ec8d3d28: Neither blocks nor log segments found. Creating new log.
I20260927 12:42:30.774528 11770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 046d11cf6e84431b8ff97750193569df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:42:30.774848 12125 log.cc:824] T 046d11cf6e84431b8ff97750193569df P 95604c169d774c85858abe84ec8d3d28: Log is configured to *not* fsync() on all Append() calls
I20260927 12:42:30.779036 12125 tablet_bootstrap.cc:492] T 046d11cf6e84431b8ff97750193569df P 95604c169d774c85858abe84ec8d3d28: No bootstrap required, opened a new log
I20260927 12:42:30.779242 12125 ts_tablet_manager.cc:1397] T 046d11cf6e84431b8ff97750193569df P 95604c169d774c85858abe84ec8d3d28: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260927 12:42:30.784404 12128 tablet_bootstrap.cc:492] T 046d11cf6e84431b8ff97750193569df P dc74971c016545978123de69450d214f: Bootstrap starting.
I20260927 12:42:30.784440 12125 raft_consensus.cc:348] T 046d11cf6e84431b8ff97750193569df P 95604c169d774c85858abe84ec8d3d28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95604c169d774c85858abe84ec8d3d28" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33735 } } peers { permanent_uuid: "f399a82a7f0c4f40950da3bf58a6279d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38131 } } peers { permanent_uuid: "dc74971c016545978123de69450d214f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39385 } }
I20260927 12:42:30.784888 12125 raft_consensus.cc:374] T 046d11cf6e84431b8ff97750193569df P 95604c169d774c85858abe84ec8d3d28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:42:30.785004 12125 raft_consensus.cc:729] T 046d11cf6e84431b8ff97750193569df P 95604c169d774c85858abe84ec8d3d28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95604c169d774c85858abe84ec8d3d28, State: Initialized, Role: FOLLOWER
I20260927 12:42:30.785610 12125 consensus_queue.cc:260] T 046d11cf6e84431b8ff97750193569df P 95604c169d774c85858abe84ec8d3d28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95604c169d774c85858abe84ec8d3d28" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33735 } } peers { permanent_uuid: "f399a82a7f0c4f40950da3bf58a6279d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38131 } } peers { permanent_uuid: "dc74971c016545978123de69450d214f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39385 } }
I20260927 12:42:30.785893 12127 tablet_bootstrap.cc:492] T 046d11cf6e84431b8ff97750193569df P f399a82a7f0c4f40950da3bf58a6279d: Bootstrap starting.
I20260927 12:42:30.786742 12128 tablet_bootstrap.cc:654] T 046d11cf6e84431b8ff97750193569df P dc74971c016545978123de69450d214f: Neither blocks nor log segments found. Creating new log.
I20260927 12:42:30.786934 12125 ts_tablet_manager.cc:1428] T 046d11cf6e84431b8ff97750193569df P 95604c169d774c85858abe84ec8d3d28: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260927 12:42:30.787349 12107 heartbeater.cc:503] Master 127.0.98.62:46199 was elected leader, sending a full tablet report...
I20260927 12:42:30.787766 12128 log.cc:824] T 046d11cf6e84431b8ff97750193569df P dc74971c016545978123de69450d214f: Log is configured to *not* fsync() on all Append() calls
I20260927 12:42:30.788237 12127 tablet_bootstrap.cc:654] T 046d11cf6e84431b8ff97750193569df P f399a82a7f0c4f40950da3bf58a6279d: Neither blocks nor log segments found. Creating new log.
I20260927 12:42:30.789083 12127 log.cc:824] T 046d11cf6e84431b8ff97750193569df P f399a82a7f0c4f40950da3bf58a6279d: Log is configured to *not* fsync() on all Append() calls
I20260927 12:42:30.792413 12127 tablet_bootstrap.cc:492] T 046d11cf6e84431b8ff97750193569df P f399a82a7f0c4f40950da3bf58a6279d: No bootstrap required, opened a new log
I20260927 12:42:30.792666 12127 ts_tablet_manager.cc:1397] T 046d11cf6e84431b8ff97750193569df P f399a82a7f0c4f40950da3bf58a6279d: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260927 12:42:30.799741 12128 tablet_bootstrap.cc:492] T 046d11cf6e84431b8ff97750193569df P dc74971c016545978123de69450d214f: No bootstrap required, opened a new log
I20260927 12:42:30.799680 12127 raft_consensus.cc:348] T 046d11cf6e84431b8ff97750193569df P f399a82a7f0c4f40950da3bf58a6279d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95604c169d774c85858abe84ec8d3d28" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33735 } } peers { permanent_uuid: "f399a82a7f0c4f40950da3bf58a6279d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38131 } } peers { permanent_uuid: "dc74971c016545978123de69450d214f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39385 } }
I20260927 12:42:30.799945 12128 ts_tablet_manager.cc:1397] T 046d11cf6e84431b8ff97750193569df P dc74971c016545978123de69450d214f: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.000s
I20260927 12:42:30.799960 12127 raft_consensus.cc:374] T 046d11cf6e84431b8ff97750193569df P f399a82a7f0c4f40950da3bf58a6279d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:42:30.800009 12127 raft_consensus.cc:729] T 046d11cf6e84431b8ff97750193569df P f399a82a7f0c4f40950da3bf58a6279d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f399a82a7f0c4f40950da3bf58a6279d, State: Initialized, Role: FOLLOWER
I20260927 12:42:30.800532 12127 consensus_queue.cc:260] T 046d11cf6e84431b8ff97750193569df P f399a82a7f0c4f40950da3bf58a6279d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95604c169d774c85858abe84ec8d3d28" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33735 } } peers { permanent_uuid: "f399a82a7f0c4f40950da3bf58a6279d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38131 } } peers { permanent_uuid: "dc74971c016545978123de69450d214f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39385 } }
I20260927 12:42:30.801683 12127 ts_tablet_manager.cc:1428] T 046d11cf6e84431b8ff97750193569df P f399a82a7f0c4f40950da3bf58a6279d: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260927 12:42:30.802030 11973 heartbeater.cc:503] Master 127.0.98.62:46199 was elected leader, sending a full tablet report...
I20260927 12:42:30.804291 12128 raft_consensus.cc:348] T 046d11cf6e84431b8ff97750193569df P dc74971c016545978123de69450d214f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95604c169d774c85858abe84ec8d3d28" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33735 } } peers { permanent_uuid: "f399a82a7f0c4f40950da3bf58a6279d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38131 } } peers { permanent_uuid: "dc74971c016545978123de69450d214f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39385 } }
I20260927 12:42:30.804534 12128 raft_consensus.cc:374] T 046d11cf6e84431b8ff97750193569df P dc74971c016545978123de69450d214f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:42:30.804582 12128 raft_consensus.cc:729] T 046d11cf6e84431b8ff97750193569df P dc74971c016545978123de69450d214f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc74971c016545978123de69450d214f, State: Initialized, Role: FOLLOWER
I20260927 12:42:30.805217 12128 consensus_queue.cc:260] T 046d11cf6e84431b8ff97750193569df P dc74971c016545978123de69450d214f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95604c169d774c85858abe84ec8d3d28" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33735 } } peers { permanent_uuid: "f399a82a7f0c4f40950da3bf58a6279d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38131 } } peers { permanent_uuid: "dc74971c016545978123de69450d214f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39385 } }
I20260927 12:42:30.806365 12128 ts_tablet_manager.cc:1428] T 046d11cf6e84431b8ff97750193569df P dc74971c016545978123de69450d214f: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 12:42:30.806734 11837 heartbeater.cc:503] Master 127.0.98.62:46199 was elected leader, sending a full tablet report...
W20260927 12:42:30.836992 11838 tablet.cc:2369] T 046d11cf6e84431b8ff97750193569df P dc74971c016545978123de69450d214f: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 12:42:30 dist-test-slave-bnc1 krb5kdc[11147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512949, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260927 12:42:30.904778 12108 tablet.cc:2369] T 046d11cf6e84431b8ff97750193569df P 95604c169d774c85858abe84ec8d3d28: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:42:30.962924 12133 raft_consensus.cc:482] T 046d11cf6e84431b8ff97750193569df P dc74971c016545978123de69450d214f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:42:30.963182 12133 raft_consensus.cc:504] T 046d11cf6e84431b8ff97750193569df P dc74971c016545978123de69450d214f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95604c169d774c85858abe84ec8d3d28" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33735 } } peers { permanent_uuid: "f399a82a7f0c4f40950da3bf58a6279d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38131 } } peers { permanent_uuid: "dc74971c016545978123de69450d214f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39385 } }
I20260927 12:42:30.964521 12133 leader_election.cc:288] T 046d11cf6e84431b8ff97750193569df P dc74971c016545978123de69450d214f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 95604c169d774c85858abe84ec8d3d28 (127.0.98.3:33735), f399a82a7f0c4f40950da3bf58a6279d (127.0.98.2:38131)
I20260927 12:42:30.971326 12120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:42:30.966493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52881 (local address 127.0.98.3:33735)
0927 12:42:30.966612 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 12:42:30.966616 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:42:30.966857 (+ 241us) server_negotiation.cc:407] Connection header received
0927 12:42:30.967052 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:42:30.967056 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:42:30.967106 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:42:30.967173 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:42:30.967581 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:42:30.968342 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:42:30.969971 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:42:30.970582 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:42:30.970667 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:42:30.970734 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 12:42:30.971102 (+ 368us) server_negotiation.cc:299] Negotiation successful
0927 12:42:30.971192 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 12:42:30.972004 12137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:42:30.964926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33735 (local address 127.0.98.1:52881)
0927 12:42:30.966772 (+ 1846us) negotiation.cc:107] Waiting for socket to connect
0927 12:42:30.966787 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:42:30.966935 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:42:30.967336 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:42:30.967339 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:42:30.967361 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:42:30.967459 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:42:30.967465 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:42:30.968507 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:42:30.968510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:42:30.969833 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:42:30.969844 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:42:30.970856 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:42:30.970859 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:42:30.970964 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:42:30.970987 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 12:42:30.971707 (+ 720us) client_negotiation.cc:240] Negotiation successful
0927 12:42:30.971784 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1777,"thread_start_us":124,"threads_started":1}
I20260927 12:42:30.972829 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "046d11cf6e84431b8ff97750193569df" candidate_uuid: "dc74971c016545978123de69450d214f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95604c169d774c85858abe84ec8d3d28" is_pre_election: true
I20260927 12:42:30.973204 12047 raft_consensus.cc:2454] T 046d11cf6e84431b8ff97750193569df P 95604c169d774c85858abe84ec8d3d28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc74971c016545978123de69450d214f in term 0.
I20260927 12:42:30.973882 11724 leader_election.cc:302] T 046d11cf6e84431b8ff97750193569df P dc74971c016545978123de69450d214f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95604c169d774c85858abe84ec8d3d28, dc74971c016545978123de69450d214f; no voters:
I20260927 12:42:30.974236 12133 raft_consensus.cc:2781] T 046d11cf6e84431b8ff97750193569df P dc74971c016545978123de69450d214f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:42:30.974316 12133 raft_consensus.cc:482] T 046d11cf6e84431b8ff97750193569df P dc74971c016545978123de69450d214f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:42:30.974363 12133 raft_consensus.cc:3037] T 046d11cf6e84431b8ff97750193569df P dc74971c016545978123de69450d214f [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:42:30.975732 12133 raft_consensus.cc:504] T 046d11cf6e84431b8ff97750193569df P dc74971c016545978123de69450d214f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95604c169d774c85858abe84ec8d3d28" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33735 } } peers { permanent_uuid: "f399a82a7f0c4f40950da3bf58a6279d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38131 } } peers { permanent_uuid: "dc74971c016545978123de69450d214f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39385 } }
I20260927 12:42:30.976138 12133 leader_election.cc:288] T 046d11cf6e84431b8ff97750193569df P dc74971c016545978123de69450d214f [CANDIDATE]: Term 1 election: Requested vote from peers 95604c169d774c85858abe84ec8d3d28 (127.0.98.3:33735), f399a82a7f0c4f40950da3bf58a6279d (127.0.98.2:38131)
I20260927 12:42:30.976926 12047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "046d11cf6e84431b8ff97750193569df" candidate_uuid: "dc74971c016545978123de69450d214f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95604c169d774c85858abe84ec8d3d28"
I20260927 12:42:30.977063 12047 raft_consensus.cc:3037] T 046d11cf6e84431b8ff97750193569df P 95604c169d774c85858abe84ec8d3d28 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:42:30.978641 12047 raft_consensus.cc:2454] T 046d11cf6e84431b8ff97750193569df P 95604c169d774c85858abe84ec8d3d28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc74971c016545978123de69450d214f in term 1.
I20260927 12:42:30.979048 11724 leader_election.cc:302] T 046d11cf6e84431b8ff97750193569df P dc74971c016545978123de69450d214f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95604c169d774c85858abe84ec8d3d28, dc74971c016545978123de69450d214f; no voters:
I20260927 12:42:30.979288 12133 raft_consensus.cc:2781] T 046d11cf6e84431b8ff97750193569df P dc74971c016545978123de69450d214f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:42:30.979503 12133 raft_consensus.cc:686] T 046d11cf6e84431b8ff97750193569df P dc74971c016545978123de69450d214f [term 1 LEADER]: Becoming Leader. State: Replica: dc74971c016545978123de69450d214f, State: Running, Role: LEADER
I20260927 12:42:30.979887 12133 consensus_queue.cc:237] T 046d11cf6e84431b8ff97750193569df P dc74971c016545978123de69450d214f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95604c169d774c85858abe84ec8d3d28" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33735 } } peers { permanent_uuid: "f399a82a7f0c4f40950da3bf58a6279d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38131 } } peers { permanent_uuid: "dc74971c016545978123de69450d214f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39385 } }
W20260927 12:42:30.982036 11974 tablet.cc:2369] T 046d11cf6e84431b8ff97750193569df P f399a82a7f0c4f40950da3bf58a6279d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:42:30.984319 11617 catalog_manager.cc:5495] T 046d11cf6e84431b8ff97750193569df P dc74971c016545978123de69450d214f reported cstate change: term changed from 0 to 1, leader changed from <none> to dc74971c016545978123de69450d214f (127.0.98.1). New cstate: current_term: 1 leader_uuid: "dc74971c016545978123de69450d214f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95604c169d774c85858abe84ec8d3d28" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f399a82a7f0c4f40950da3bf58a6279d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc74971c016545978123de69450d214f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39385 } health_report { overall_health: HEALTHY } } }
I20260927 12:42:30.987206 12122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:42:30.966269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38691 (local address 127.0.98.2:38131)
0927 12:42:30.966391 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 12:42:30.966396 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:42:30.970146 (+ 3750us) server_negotiation.cc:407] Connection header received
0927 12:42:30.970363 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:42:30.970366 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:42:30.970416 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:42:30.970463 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:42:30.970874 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:42:30.971635 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:42:30.985746 (+ 14111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:42:30.986440 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:42:30.986536 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:42:30.986600 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 12:42:30.986978 (+ 378us) server_negotiation.cc:299] Negotiation successful
0927 12:42:30.987061 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 12:42:30.987814 12136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:42:30.965774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38131 (local address 127.0.98.1:38691)
0927 12:42:30.970035 (+ 4261us) negotiation.cc:107] Waiting for socket to connect
0927 12:42:30.970051 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 12:42:30.970217 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:42:30.970615 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:42:30.970618 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:42:30.970634 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:42:30.970744 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:42:30.970750 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:42:30.971797 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:42:30.971800 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:42:30.972972 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:42:30.972982 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:42:30.986708 (+ 13726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:42:30.986712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:42:30.986829 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:42:30.986854 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 12:42:30.987579 (+ 725us) client_negotiation.cc:240] Negotiation successful
0927 12:42:30.987669 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4191,"thread_start_us":77,"threads_started":1}
I20260927 12:42:30.988634 11927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "046d11cf6e84431b8ff97750193569df" candidate_uuid: "dc74971c016545978123de69450d214f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f399a82a7f0c4f40950da3bf58a6279d" is_pre_election: true
I20260927 12:42:30.988960 11927 raft_consensus.cc:2454] T 046d11cf6e84431b8ff97750193569df P f399a82a7f0c4f40950da3bf58a6279d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc74971c016545978123de69450d214f in term 0.
I20260927 12:42:30.989317 11926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "046d11cf6e84431b8ff97750193569df" candidate_uuid: "dc74971c016545978123de69450d214f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f399a82a7f0c4f40950da3bf58a6279d"
I20260927 12:42:30.989439 11926 raft_consensus.cc:3037] T 046d11cf6e84431b8ff97750193569df P f399a82a7f0c4f40950da3bf58a6279d [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:42:30.990856 11926 raft_consensus.cc:2454] T 046d11cf6e84431b8ff97750193569df P f399a82a7f0c4f40950da3bf58a6279d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc74971c016545978123de69450d214f in term 1.
I20260927 12:42:31.030725 12124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:42:31.025438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52580 (local address 127.0.98.1:39385)
0927 12:42:31.025625 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 12:42:31.025630 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:42:31.025649 (+ 19us) server_negotiation.cc:407] Connection header received
0927 12:42:31.025761 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:42:31.025764 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:42:31.025842 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:42:31.025961 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:42:31.026393 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:42:31.027158 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:42:31.028499 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:42:31.028853 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:42:31.028952 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:42:31.029389 (+ 437us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:42:31.029970 (+ 581us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:42:31.030088 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 12:42:31.030501 (+ 413us) server_negotiation.cc:299] Negotiation successful
0927 12:42:31.030563 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 12:42:31.038337 11926 raft_consensus.cc:1261] T 046d11cf6e84431b8ff97750193569df P f399a82a7f0c4f40950da3bf58a6279d [term 1 FOLLOWER]: Refusing update from remote peer dc74971c016545978123de69450d214f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:42:31.039477 12138 consensus_queue.cc:1035] T 046d11cf6e84431b8ff97750193569df P dc74971c016545978123de69450d214f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f399a82a7f0c4f40950da3bf58a6279d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:42:31.046625 12047 raft_consensus.cc:1261] T 046d11cf6e84431b8ff97750193569df P 95604c169d774c85858abe84ec8d3d28 [term 1 FOLLOWER]: Refusing update from remote peer dc74971c016545978123de69450d214f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:42:31.047446 12138 consensus_queue.cc:1035] T 046d11cf6e84431b8ff97750193569df P dc74971c016545978123de69450d214f [LEADER]: Connected to new peer: Peer: permanent_uuid: "95604c169d774c85858abe84ec8d3d28" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:42:31.077013 12144 mvcc.cc:205] Tried to move back new op lower bound from 7333941047443406848 to 7333941047217467392. Current Snapshot: MvccSnapshot[applied={T|T < 7333941047443406848}]
I20260927 12:42:31.079206 11617 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:42:31.079375 11617 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:42:31.084041 11617 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6a85f0a52e104948a1c5748817fd2878: Sending DeleteTablet for 3 replicas of tablet 046d11cf6e84431b8ff97750193569df
I20260927 12:42:31.085212 11770 tablet_service.cc:1514] Processing DeleteTablet for tablet 046d11cf6e84431b8ff97750193569df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:42:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52568
I20260927 12:42:31.085284 12041 tablet_service.cc:1514] Processing DeleteTablet for tablet 046d11cf6e84431b8ff97750193569df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:42:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49884
I20260927 12:42:31.086077 11907 tablet_service.cc:1514] Processing DeleteTablet for tablet 046d11cf6e84431b8ff97750193569df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:42:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34600
I20260927 12:42:31.087822 12142 mvcc.cc:205] Tried to move back new op lower bound from 7333941047443406848 to 7333941047217467392. Current Snapshot: MvccSnapshot[applied={T|T < 7333941047443406848}]
I20260927 12:42:31.089043 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 11704
I20260927 12:42:31.093776 12151 tablet_replica.cc:330] T 046d11cf6e84431b8ff97750193569df P f399a82a7f0c4f40950da3bf58a6279d: stopping tablet replica
I20260927 12:42:31.094063 12149 tablet_replica.cc:330] T 046d11cf6e84431b8ff97750193569df P 95604c169d774c85858abe84ec8d3d28: stopping tablet replica
I20260927 12:42:31.094483 12151 raft_consensus.cc:2229] T 046d11cf6e84431b8ff97750193569df P f399a82a7f0c4f40950da3bf58a6279d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:42:31.094841 12149 raft_consensus.cc:2229] T 046d11cf6e84431b8ff97750193569df P 95604c169d774c85858abe84ec8d3d28 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:42:31.095187 12151 raft_consensus.cc:2258] T 046d11cf6e84431b8ff97750193569df P f399a82a7f0c4f40950da3bf58a6279d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:42:31.095520 12149 raft_consensus.cc:2258] T 046d11cf6e84431b8ff97750193569df P 95604c169d774c85858abe84ec8d3d28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:42:31.097410 12149 ts_tablet_manager.cc:1905] T 046d11cf6e84431b8ff97750193569df P 95604c169d774c85858abe84ec8d3d28: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:42:31.100466 12149 ts_tablet_manager.cc:1918] T 046d11cf6e84431b8ff97750193569df P 95604c169d774c85858abe84ec8d3d28: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:42:31.100673 12149 log.cc:1196] T 046d11cf6e84431b8ff97750193569df P 95604c169d774c85858abe84ec8d3d28: Deleting WAL directory at /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/ts-2/wal/wals/046d11cf6e84431b8ff97750193569df
I20260927 12:42:31.101120 12149 ts_tablet_manager.cc:1939] T 046d11cf6e84431b8ff97750193569df P 95604c169d774c85858abe84ec8d3d28: Deleting consensus metadata
I20260927 12:42:31.102959 12151 ts_tablet_manager.cc:1905] T 046d11cf6e84431b8ff97750193569df P f399a82a7f0c4f40950da3bf58a6279d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:42:31.105638 12151 ts_tablet_manager.cc:1918] T 046d11cf6e84431b8ff97750193569df P f399a82a7f0c4f40950da3bf58a6279d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:42:31.105818 12151 log.cc:1196] T 046d11cf6e84431b8ff97750193569df P f399a82a7f0c4f40950da3bf58a6279d: Deleting WAL directory at /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790512840836255-392-0/minicluster-data/ts-1/wal/wals/046d11cf6e84431b8ff97750193569df
I20260927 12:42:31.106179 12151 ts_tablet_manager.cc:1939] T 046d11cf6e84431b8ff97750193569df P f399a82a7f0c4f40950da3bf58a6279d: Deleting consensus metadata
W20260927 12:42:31.107061 11610 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39385 (error 108)
I20260927 12:42:31.108189 12123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:42:31.107857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39385 (local address 127.0.0.1:52586)
0927 12:42:31.108036 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0927 12:42:31.108101 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
W20260927 12:42:31.108462 11610 catalog_manager.cc:4584] TS dc74971c016545978123de69450d214f (127.0.98.1:39385): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 046d11cf6e84431b8ff97750193569df: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39385: connect: Connection refused (error 111)
I20260927 12:42:31.110174 11607 catalog_manager.cc:4841] TS 95604c169d774c85858abe84ec8d3d28 (127.0.98.3:33735): tablet 046d11cf6e84431b8ff97750193569df (table test-table [id=94f1ccb7c0af4e63b885fc72cb2effc8]) successfully deleted
I20260927 12:42:31.111359 11608 catalog_manager.cc:4841] TS f399a82a7f0c4f40950da3bf58a6279d (127.0.98.2:38131): tablet 046d11cf6e84431b8ff97750193569df (table test-table [id=94f1ccb7c0af4e63b885fc72cb2effc8]) successfully deleted
I20260927 12:42:31.111886 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 11843
I20260927 12:42:31.122398 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 11978
I20260927 12:42:31.130448 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 11591
I20260927 12:42:31.136682 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 12:42:31.472084 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 12:42:31.472227 392 mini_ranger.cc:66] Stopping Ranger...
I20260927 12:42:31.816680 392 mini_ranger.cc:68] Stopped Ranger
2026-09-27 12:42:31.816 UTC [11177] LOG: received smart shutdown request
2026-09-27 12:42:31.820 UTC [11177] LOG: background worker "logical replication launcher" (PID 11186) exited with exit code 1
2026-09-27 12:42:31.820 UTC [11181] LOG: shutting down
2026-09-27 12:42:31.997 UTC [11177] LOG: database system is shut down
2026-09-27T12:42:31Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46769 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:42:32 dist-test-slave-bnc1 krb5kdc[12162](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:42:32 dist-test-slave-bnc1 krb5kdc[12162](info): set up 2 sockets
Sep 27 12:42:32 dist-test-slave-bnc1 krb5kdc[12162](info): commencing operation
krb5kdc: starting...
W20260927 12:42:33.154003 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/krb5kdc/test-user.keytab.
Sep 27 12:42:33 dist-test-slave-bnc1 krb5kdc[12162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512953, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:42:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:42:33Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260927 12:42:33.320557 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task_Q2zcw/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/postgres -l logfile start
2026-09-27 12:42:35.819 UTC [12192] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 12:42:35.819 UTC [12192] LOG: listening on IPv4 address "127.0.98.20", port 46231
2026-09-27 12:42:35.822 UTC [12192] LOG: listening on Unix socket "/tmp/.s.PGSQL.46231"
2026-09-27 12:42:35.843 UTC [12195] LOG: database system was shut down at 2026-09-27 12:42:34 UTC
2026-09-27 12:42:35.847 UTC [12192] LOG: database system is ready to accept connections
I20260927 12:42:36.813391 392 mini_postgres.cc:96] Postgres bound to 46231
2026-09-27 12:42:36.819 UTC [12203] FATAL: database "slave" does not exist
127.0.98.20:46231 - accepting connections
I20260927 12:42:36.820281 392 mini_ranger.cc:161] Starting Ranger...
I20260927 12:42:36.834842 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 12:42:37.320757 392 mini_ranger.cc:87] Created ranger Postgres database
I20260927 12:42:37.320845 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ranger-admin
2026-09-27 12:42:37,599 [I] DB FLAVOR :POSTGRES
2026-09-27 12:42:37,600 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 12:42:37,600 [I] Checking connection..
2026-09-27 12:42:37,600 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_Q2zcw/build/debug/bin/postgresql.jar:/tmp/dist-test-task_Q2zcw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 12:42:37,837 [I] Checking connection passed.
2026-09-27 12:42:37,837 [I] --------- Verifying version history table ---------
2026-09-27 12:42:37,837 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_Q2zcw/build/debug/bin/postgresql.jar:/tmp/dist-test-task_Q2zcw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 12:42:38,063 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 12:42:38,063 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_Q2zcw/build/debug/bin/postgresql.jar:/tmp/dist-test-task_Q2zcw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 12:42:38,292 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 12:42:38,292 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 12:42:38,293 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_Q2zcw/build/debug/bin/postgresql.jar:/tmp/dist-test-task_Q2zcw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46231/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task_Q2zcw/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 12:42:38.545 UTC [12267] WARNING: there is no transaction in progress
2026-09-27 12:42:38,551 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 12:42:38,551 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_Q2zcw/build/debug/bin/postgresql.jar:/tmp/dist-test-task_Q2zcw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 12:42:38,830 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 12:42:38,830 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 12:42:38,831 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_Q2zcw/build/debug/bin/postgresql.jar:/tmp/dist-test-task_Q2zcw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 12:42:39,067 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_Q2zcw/build/debug/bin/postgresql.jar:/tmp/dist-test-task_Q2zcw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 12:42:39,324 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_Q2zcw/build/debug/bin/postgresql.jar:/tmp/dist-test-task_Q2zcw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-bnc1.c.gcp-upstream.internal','N') ;"
2026-09-27 12:42:39,571 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_Q2zcw/build/debug/bin/postgresql.jar:/tmp/dist-test-task_Q2zcw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 12:42:39,831 [I] Table x_portal_user does not exist in database ranger
2026-09-27 12:42:39,832 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_Q2zcw/build/debug/bin/postgresql.jar:/tmp/dist-test-task_Q2zcw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 12:42:40,061 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 12:42:40,061 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_Q2zcw/build/debug/bin/postgresql.jar:/tmp/dist-test-task_Q2zcw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 12:42:40,296 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_Q2zcw/build/debug/bin/postgresql.jar:/tmp/dist-test-task_Q2zcw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 12:42:40,524 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 12:42:40,524 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_Q2zcw/build/debug/bin/postgresql.jar:/tmp/dist-test-task_Q2zcw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46231/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task_Q2zcw/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 12:42:41.496 UTC [12375] WARNING: there is no transaction in progress
2026-09-27 12:42:41.512 UTC [12375] WARNING: there is no transaction in progress
2026-09-27 12:42:41.528 UTC [12375] WARNING: there is no transaction in progress
2026-09-27 12:42:41.543 UTC [12375] WARNING: there is no transaction in progress
2026-09-27 12:42:41.562 UTC [12375] WARNING: there is no transaction in progress
2026-09-27 12:42:41.579 UTC [12375] WARNING: there is no transaction in progress
2026-09-27 12:42:41.636 UTC [12375] WARNING: there is no transaction in progress
2026-09-27 12:42:41.644 UTC [12375] WARNING: there is no transaction in progress
2026-09-27 12:42:41.659 UTC [12375] WARNING: there is no transaction in progress
2026-09-27 12:42:41.669 UTC [12375] WARNING: there is no transaction in progress
2026-09-27 12:42:41.680 UTC [12375] WARNING: there is no transaction in progress
2026-09-27 12:42:41.695 UTC [12375] WARNING: there is no transaction in progress
2026-09-27 12:42:41.706 UTC [12375] WARNING: there is no transaction in progress
2026-09-27 12:42:41.713 UTC [12375] WARNING: there is no transaction in progress
2026-09-27 12:42:41.721 UTC [12375] WARNING: there is no transaction in progress
2026-09-27 12:42:41.728 UTC [12375] WARNING: there is no transaction in progress
2026-09-27 12:42:42,392 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 12:42:42,392 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_Q2zcw/build/debug/bin/postgresql.jar:/tmp/dist-test-task_Q2zcw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 12:42:42,647 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 12:42:42,648 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_Q2zcw/build/debug/bin/postgresql.jar:/tmp/dist-test-task_Q2zcw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 12:42:42,873 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 12:42:42,873 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_Q2zcw/build/debug/bin/postgresql.jar:/tmp/dist-test-task_Q2zcw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 12:42:43,113 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 12:42:43,114 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_Q2zcw/build/debug/bin/postgresql.jar:/tmp/dist-test-task_Q2zcw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 12:42:43,353 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_Q2zcw/build/debug/bin/postgresql.jar:/tmp/dist-test-task_Q2zcw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 12:42:43,592 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_Q2zcw/build/debug/bin/postgresql.jar:/tmp/dist-test-task_Q2zcw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-bnc1.c.gcp-upstream.internal';"
2026-09-27 12:42:43,811 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 12:42:43,812 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_Q2zcw/build/debug/bin/postgresql.jar:/tmp/dist-test-task_Q2zcw/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46231/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 12:42:44,050 [I] DB_PATCHES have already been applied
I20260927 12:42:44.056348 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ranger-admin:/tmp/dist-test-task_Q2zcw/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task_Q2zcw/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task_Q2zcw/build/debug/bin/postgresql.jar:/tmp/dist-test-task_Q2zcw/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 12:42:44.056411 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260927 12:42:44.059734 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:34339
Sep 27, 2026 12:42:44 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 12:42:44 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 12:42:44 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 12:42:44 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 12:42:44 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 12:42:44 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 12:42:44 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 12:42:45.166381 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.346s user 0.001s sys 0.008s
Sep 27, 2026 12:43:05 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 12:43:05 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 27, 2026 12:43:05 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 12:43:05 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 12:43:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 12:43:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 12:43:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 12:43:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 12:43:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 12:43:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 12:43:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 12:43:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 12:43:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 12:43:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 12:43:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 12:43:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 12:43:05 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 12:43:06 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 12:43:06.833283 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 12:43:06 dist-test-slave-bnc1 krb5kdc[12162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512986, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 12:43:06.916673 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 12:43:06.928224 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 12:43:07.798556 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 12:43:07.798643 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ranger-kms
I20260927 12:43:07.798666 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:46231
2026-09-27 12:43:07,965 [I] DB FLAVOR :POSTGRES
2026-09-27 12:43:07,965 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 12:43:07,965 [I] Checking connection
2026-09-27 12:43:07,965 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_Q2zcw/build/debug/bin/postgresql.jar:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46231/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 12:43:08,212 [I] connection success
2026-09-27 12:43:08,212 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 12:43:08,212 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 12:43:08,212 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_Q2zcw/build/debug/bin/postgresql.jar:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46231/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 12:43:08,441 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 12:43:08,441 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 12:43:08,442 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 12:43:08,442 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_Q2zcw/build/debug/bin/postgresql.jar:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:46231/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 12:43:08,692 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 12:43:09.175808 392 mini_ranger_kms.cc:310] Created kms service
I20260927 12:43:09.420059 392 mini_ranger_kms.cc:326] Created kudu user
I20260927 12:43:09.461408 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 12:43:09 dist-test-slave-bnc1 krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512986, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 12:43:09.900394 392 mini_ranger_kms.cc:384] Added ranger policy
I20260927 12:43:09.900556 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ranger-kms:/tmp/dist-test-task_Q2zcw/build/debug/bin/postgresql.jar:/tmp/dist-test-task_Q2zcw/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task_Q2zcw/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 12:43:09.900579 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260927 12:43:09.904232 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12555
I20260927 12:43:09.904310 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:59175
Sep 27, 2026 12:43:10 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 12:43:10 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 12:43:10 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 12:43:10 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 12:43:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 12:43:11 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 12:43:12.137107 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.220s user 0.008s sys 0.143s
I20260927 12:43:12.137234 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 12:43:12.137319 392 mini_ranger_kms.cc:401] 127.0.98.20:59175/kms/v1/keys
Sep 27 12:43:12 dist-test-slave-bnc1 krb5kdc[12162](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790512992, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 12:43:12 dist-test-slave-bnc1 krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512992, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 12:43:12 dist-test-slave-bnc1 krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512992, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 27 12:43:15 dist-test-slave-bnc1 krb5kdc[12162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:43:15.152966 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40293
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40293
--ranger_config_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:59175/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:43:15.286572 12606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:43:15.286810 12606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:43:15.286856 12606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:43:15.290809 12606 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:43:15.290865 12606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:43:15.290884 12606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:43:15.290900 12606 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:43:15.290915 12606 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:43:15.296092 12606 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:59175/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40293
--ranger_config_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12606
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:43:15.297057 12606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:43:15.297998 12606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:43:15.306600 12614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:43:15.306692 12612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:43:15.306749 12611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:43:15.306622 12606 server_base.cc:1034] running on GCE node
I20260927 12:43:15.307515 12606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:43:15.308562 12606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:43:15.309744 12606 hybrid_clock.cc:648] HybridClock initialized: now 1790512995309720 us; error 43 us; skew 500 ppm
Sep 27 12:43:15 dist-test-slave-bnc1 krb5kdc[12162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512995, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:43:15.312352 12606 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:43:15.313242 12606 webserver.cc:460] Webserver started at http://127.0.98.62:39805/ using document root <none> and password file <none>
I20260927 12:43:15.313849 12606 fs_manager.cc:359] Metadata directory not provided
I20260927 12:43:15.313952 12606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:43:15.314167 12606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 12:43:15 dist-test-slave-bnc1 krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512995, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 12:43:15.376938 12606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/master-0/data/instance:
uuid: "38a6c6e565764580ac353d604b8b057b"
format_stamp: "Formatted at 2026-09-27 12:43:15 on dist-test-slave-bnc1"
server_key: "aa1d2f37ff60e9762a6174b0832ee63c"
server_key_iv: "95ebdd2906d5c6b8234b799ac832d006"
server_key_version: "kuduclusterkey@0"
I20260927 12:43:15.377676 12606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/master-0/wal/instance:
uuid: "38a6c6e565764580ac353d604b8b057b"
format_stamp: "Formatted at 2026-09-27 12:43:15 on dist-test-slave-bnc1"
server_key: "aa1d2f37ff60e9762a6174b0832ee63c"
server_key_iv: "95ebdd2906d5c6b8234b799ac832d006"
server_key_version: "kuduclusterkey@0"
I20260927 12:43:15.381904 12606 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 12:43:15.399549 12621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:43:15.401048 12606 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 12:43:15.401197 12606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/master-0/wal
uuid: "38a6c6e565764580ac353d604b8b057b"
format_stamp: "Formatted at 2026-09-27 12:43:15 on dist-test-slave-bnc1"
server_key: "aa1d2f37ff60e9762a6174b0832ee63c"
server_key_iv: "95ebdd2906d5c6b8234b799ac832d006"
server_key_version: "kuduclusterkey@0"
I20260927 12:43:15.401331 12606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:43:15.422184 12606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:43:15.425503 12606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:43:15.425774 12606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:43:15.434011 12606 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40293
I20260927 12:43:15.434020 12673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40293 every 8 connection(s)
I20260927 12:43:15.435040 12606 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:43:15.437918 12674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:43:15.441283 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 12606
I20260927 12:43:15.441411 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/master-0/wal/instance
I20260927 12:43:15.441772 392 external_mini_cluster.cc:1428] Setting key 8037051dd54ac35c004b5e9aa904cc16
I20260927 12:43:15.444217 12674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38a6c6e565764580ac353d604b8b057b: Bootstrap starting.
I20260927 12:43:15.446800 12674 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 38a6c6e565764580ac353d604b8b057b: Neither blocks nor log segments found. Creating new log.
I20260927 12:43:15.447659 12674 log.cc:824] T 00000000000000000000000000000000 P 38a6c6e565764580ac353d604b8b057b: Log is configured to *not* fsync() on all Append() calls
I20260927 12:43:15.449843 12674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38a6c6e565764580ac353d604b8b057b: No bootstrap required, opened a new log
Sep 27 12:43:15 dist-test-slave-bnc1 krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:43:15.454200 12674 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38a6c6e565764580ac353d604b8b057b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38a6c6e565764580ac353d604b8b057b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40293 } }
I20260927 12:43:15.454504 12674 raft_consensus.cc:374] T 00000000000000000000000000000000 P 38a6c6e565764580ac353d604b8b057b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:43:15.454589 12674 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38a6c6e565764580ac353d604b8b057b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38a6c6e565764580ac353d604b8b057b, State: Initialized, Role: FOLLOWER
I20260927 12:43:15.455054 12674 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38a6c6e565764580ac353d604b8b057b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38a6c6e565764580ac353d604b8b057b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40293 } }
I20260927 12:43:15.455225 12674 raft_consensus.cc:388] T 00000000000000000000000000000000 P 38a6c6e565764580ac353d604b8b057b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:43:15.455318 12674 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38a6c6e565764580ac353d604b8b057b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:43:15.455449 12674 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38a6c6e565764580ac353d604b8b057b [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:43:15.456584 12674 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38a6c6e565764580ac353d604b8b057b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38a6c6e565764580ac353d604b8b057b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40293 } }
I20260927 12:43:15.456956 12674 leader_election.cc:302] T 00000000000000000000000000000000 P 38a6c6e565764580ac353d604b8b057b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38a6c6e565764580ac353d604b8b057b; no voters:
I20260927 12:43:15.457305 12674 leader_election.cc:288] T 00000000000000000000000000000000 P 38a6c6e565764580ac353d604b8b057b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:43:15.457381 12679 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38a6c6e565764580ac353d604b8b057b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:43:15.457643 12679 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38a6c6e565764580ac353d604b8b057b [term 1 LEADER]: Becoming Leader. State: Replica: 38a6c6e565764580ac353d604b8b057b, State: Running, Role: LEADER
I20260927 12:43:15.457993 12679 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38a6c6e565764580ac353d604b8b057b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38a6c6e565764580ac353d604b8b057b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40293 } }
I20260927 12:43:15.458691 12674 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38a6c6e565764580ac353d604b8b057b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:43:15.459213 12679 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38a6c6e565764580ac353d604b8b057b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38a6c6e565764580ac353d604b8b057b. Latest consensus state: current_term: 1 leader_uuid: "38a6c6e565764580ac353d604b8b057b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38a6c6e565764580ac353d604b8b057b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40293 } } }
I20260927 12:43:15.459399 12679 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38a6c6e565764580ac353d604b8b057b [sys.catalog]: This master's current role is: LEADER
I20260927 12:43:15.459565 12677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:15.443660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41712 (local address 127.0.98.62:40293)
0927 12:43:15.444187 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0927 12:43:15.444210 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 12:43:15.444253 (+ 43us) server_negotiation.cc:407] Connection header received
0927 12:43:15.445068 (+ 815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:43:15.445091 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:43:15.445523 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:43:15.445906 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:43:15.446768 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:15.448188 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:15.449360 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:15.450181 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:15.450343 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:15.453480 (+ 3137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:43:15.453487 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:43:15.453503 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:43:15.453534 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:43:15.455683 (+ 2149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:15.456193 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:15.456199 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:15.456205 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:15.456339 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:15.456686 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:15.456689 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:15.456691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:15.457199 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:43:15.458119 (+ 920us) server_negotiation.cc:1035] Waiting for connection context
0927 12:43:15.458267 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 12:43:15.458566 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":111,"threads_started":1}
I20260927 12:43:15.459775 12679 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38a6c6e565764580ac353d604b8b057b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "38a6c6e565764580ac353d604b8b057b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38a6c6e565764580ac353d604b8b057b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40293 } } }
I20260927 12:43:15.459854 12679 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38a6c6e565764580ac353d604b8b057b [sys.catalog]: This master's current role is: LEADER
I20260927 12:43:15.461304 12674 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 12:43:15.465801 12685 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:43:15.471401 12685 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:43:15.477945 12685 catalog_manager.cc:1347] Generated new cluster ID: df8f3e8b28034e3a84b93c084799d0f2
I20260927 12:43:15.478019 12685 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:43:15.484179 12685 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:43:15.485155 12685 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:43:15.493613 12685 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 38a6c6e565764580ac353d604b8b057b: Generated new TSK 0
I20260927 12:43:15.494354 12685 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 12:43:16.197230 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:59175/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40293
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:43:16.312448 12719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:43:16.312764 12719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:43:16.312886 12719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:43:16.317183 12719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:43:16.317255 12719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:43:16.317384 12719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:43:16.323709 12719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:59175/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:43:16.324810 12719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:43:16.326100 12719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:43:16.335975 12727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:43:16.336038 12725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:43:16.335994 12724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:43:16.336268 12719 server_base.cc:1034] running on GCE node
I20260927 12:43:16.336810 12719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:43:16.337610 12719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:43:16.338802 12719 hybrid_clock.cc:648] HybridClock initialized: now 1790512996338763 us; error 46 us; skew 500 ppm
Sep 27 12:43:16 dist-test-slave-bnc1 krb5kdc[12162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512996, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:43:16.346042 12719 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 12:43:16.347085 12719 webserver.cc:460] Webserver started at http://127.0.98.1:41689/ using document root <none> and password file <none>
I20260927 12:43:16.347680 12719 fs_manager.cc:359] Metadata directory not provided
I20260927 12:43:16.347772 12719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:43:16.348006 12719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 12:43:16 dist-test-slave-bnc1 krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512996, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 12:43:16.366243 12719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-0/data/instance:
uuid: "3f555fa910234b098329094d7fee0dfb"
format_stamp: "Formatted at 2026-09-27 12:43:16 on dist-test-slave-bnc1"
server_key: "4e81ea75998c50963c31a30350b48219"
server_key_iv: "3e6ff185ca4806357b1536d03ad35103"
server_key_version: "kuduclusterkey@0"
I20260927 12:43:16.366858 12719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3f555fa910234b098329094d7fee0dfb"
format_stamp: "Formatted at 2026-09-27 12:43:16 on dist-test-slave-bnc1"
server_key: "4e81ea75998c50963c31a30350b48219"
server_key_iv: "3e6ff185ca4806357b1536d03ad35103"
server_key_version: "kuduclusterkey@0"
I20260927 12:43:16.370843 12719 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 12:43:16.389982 12736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:43:16.391746 12719 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260927 12:43:16.391886 12719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-0/wal
uuid: "3f555fa910234b098329094d7fee0dfb"
format_stamp: "Formatted at 2026-09-27 12:43:16 on dist-test-slave-bnc1"
server_key: "4e81ea75998c50963c31a30350b48219"
server_key_iv: "3e6ff185ca4806357b1536d03ad35103"
server_key_version: "kuduclusterkey@0"
I20260927 12:43:16.392031 12719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:43:16.409973 12719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:43:16.413712 12719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:43:16.413899 12719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:43:16.414566 12719 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:43:16.415555 12719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:43:16.415612 12719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:43:16.415663 12719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:43:16.415722 12719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:43:16.426133 12719 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37985
I20260927 12:43:16.426194 12849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37985 every 8 connection(s)
I20260927 12:43:16.427045 12719 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:43:16.427685 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 12719
I20260927 12:43:16.427805 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-0/wal/instance
I20260927 12:43:16.428134 392 external_mini_cluster.cc:1428] Setting key 64abc05fb3a67abc161b89297a9ea833
Sep 27 12:43:16 dist-test-slave-bnc1 krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512996, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 12:43:16.444489 12852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:16.430086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33749 (local address 127.0.98.62:40293)
0927 12:43:16.431696 (+ 1610us) server_negotiation.cc:206] Beginning negotiation
0927 12:43:16.431705 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 12:43:16.431723 (+ 18us) server_negotiation.cc:407] Connection header received
0927 12:43:16.432933 (+ 1210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:43:16.432936 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:43:16.432987 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:43:16.433087 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:43:16.434745 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:16.435367 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:16.436499 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:16.436736 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:16.436846 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:16.439183 (+ 2337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:43:16.439194 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:43:16.439196 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:43:16.439216 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:43:16.440894 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:16.441402 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:16.441405 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:16.441407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:16.441491 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:16.441893 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:16.441896 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:16.441898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:16.443400 (+ 1502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:43:16.443510 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 12:43:16.444186 (+ 676us) server_negotiation.cc:299] Negotiation successful
0927 12:43:16.444312 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1523,"thread_start_us":113,"threads_started":1}
I20260927 12:43:16.452118 12853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:16.430553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40293 (local address 127.0.98.1:33749)
0927 12:43:16.431613 (+ 1060us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:16.431655 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 12:43:16.432595 (+ 940us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:43:16.433390 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:43:16.433399 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:43:16.433897 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:43:16.434547 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:16.434571 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:16.435495 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:16.435499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:16.436375 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:16.436384 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:16.436880 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:16.436884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:16.437063 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:16.437680 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:43:16.437696 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:43:16.438991 (+ 1295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:43:16.441022 (+ 2031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:43:16.441031 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:43:16.441045 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:43:16.441306 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:43:16.441606 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:43:16.441609 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:43:16.441611 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:43:16.441794 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:43:16.443537 (+ 1743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:43:16.443545 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:43:16.443882 (+ 337us) client_negotiation.cc:769] Sending connection context
0927 12:43:16.444096 (+ 214us) client_negotiation.cc:240] Negotiation successful
0927 12:43:16.444422 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":880,"thread_start_us":127,"threads_started":1}
I20260927 12:43:16.453783 12850 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40293
I20260927 12:43:16.454061 12850 heartbeater.cc:464] Registering TS with master...
I20260927 12:43:16.454610 12850 heartbeater.cc:511] Master 127.0.98.62:40293 requested a full tablet report, sending...
I20260927 12:43:16.456285 12638 ts_manager.cc:194] Registered new tserver with Master: 3f555fa910234b098329094d7fee0dfb (127.0.98.1:37985)
I20260927 12:43:16.457647 12638 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33749
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 12:43:16.460449 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:59175/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40293
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 12:43:16 dist-test-slave-bnc1 krb5kdc[12162](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790512996, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 12:43:16.591605 12856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:43:16.592026 12856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:43:16.592129 12856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:43:16.598028 12856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:43:16.598173 12856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:43:16.598301 12856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:43:16.606071 12856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:59175/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:43:16.607234 12856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:43:16.608475 12856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:43:16.620183 12866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:43:16.620261 12863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:43:16.620504 12864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:43:16.621265 12856 server_base.cc:1034] running on GCE node
I20260927 12:43:16.621763 12856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:43:16.622457 12856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:43:16.637981 12856 hybrid_clock.cc:648] HybridClock initialized: now 1790512996637930 us; error 72 us; skew 500 ppm
Sep 27 12:43:16 dist-test-slave-bnc1 krb5kdc[12162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512996, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:43:16.641065 12856 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 12:43:16.642238 12856 webserver.cc:460] Webserver started at http://127.0.98.2:39649/ using document root <none> and password file <none>
I20260927 12:43:16.642946 12856 fs_manager.cc:359] Metadata directory not provided
I20260927 12:43:16.643097 12856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:43:16.643339 12856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 12:43:16 dist-test-slave-bnc1 krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512996, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 12:43:16.660071 12856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-1/data/instance:
uuid: "93287b8840454786a4944835236be293"
format_stamp: "Formatted at 2026-09-27 12:43:16 on dist-test-slave-bnc1"
server_key: "094ce18f68ed1f915dbbbe5afb0f5124"
server_key_iv: "1f716075cf969b96fdead3a6fe962892"
server_key_version: "kuduclusterkey@0"
I20260927 12:43:16.660809 12856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-1/wal/instance:
uuid: "93287b8840454786a4944835236be293"
format_stamp: "Formatted at 2026-09-27 12:43:16 on dist-test-slave-bnc1"
server_key: "094ce18f68ed1f915dbbbe5afb0f5124"
server_key_iv: "1f716075cf969b96fdead3a6fe962892"
server_key_version: "kuduclusterkey@0"
I20260927 12:43:16.665542 12856 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 12:43:16.681468 12874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:43:16.683200 12856 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 12:43:16.683336 12856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-1/wal
uuid: "93287b8840454786a4944835236be293"
format_stamp: "Formatted at 2026-09-27 12:43:16 on dist-test-slave-bnc1"
server_key: "094ce18f68ed1f915dbbbe5afb0f5124"
server_key_iv: "1f716075cf969b96fdead3a6fe962892"
server_key_version: "kuduclusterkey@0"
I20260927 12:43:16.683477 12856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:43:16.701861 12856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:43:16.705461 12856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:43:16.705740 12856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:43:16.706504 12856 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:43:16.707616 12856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:43:16.707729 12856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:43:16.707828 12856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:43:16.707896 12856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:43:16.721987 12856 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38825
I20260927 12:43:16.723318 12856 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:43:16.723981 12987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38825 every 8 connection(s)
I20260927 12:43:16.728025 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 12856
I20260927 12:43:16.728150 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-1/wal/instance
I20260927 12:43:16.728441 392 external_mini_cluster.cc:1428] Setting key 2366cba542c735bb77919470d1257b0e
Sep 27 12:43:16 dist-test-slave-bnc1 krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512996, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:43:16.746930 12852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:16.727936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55975 (local address 127.0.98.62:40293)
0927 12:43:16.728070 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 12:43:16.728074 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:43:16.728867 (+ 793us) server_negotiation.cc:407] Connection header received
0927 12:43:16.731956 (+ 3089us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:43:16.731959 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:43:16.732011 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:43:16.732065 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:43:16.733767 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:16.734571 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:16.736262 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:16.736615 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:16.736694 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:16.742424 (+ 5730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:43:16.742428 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:43:16.742431 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:43:16.742461 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:43:16.744355 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:16.745028 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:16.745033 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:16.745035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:16.745124 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:16.745582 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:16.745586 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:16.745588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:16.745806 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:43:16.745894 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 12:43:16.746602 (+ 708us) server_negotiation.cc:299] Negotiation successful
0927 12:43:16.746737 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 12:43:16.748891 12990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:16.726815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40293 (local address 127.0.98.2:55975)
0927 12:43:16.728701 (+ 1886us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:16.728758 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 12:43:16.730958 (+ 2200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:43:16.732469 (+ 1511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:43:16.732479 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:43:16.732845 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:43:16.733581 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:16.733606 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:16.734727 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:16.734730 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:16.736122 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:16.736133 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:16.736897 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:16.736901 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:16.737086 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:16.740279 (+ 3193us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:43:16.740300 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:43:16.742220 (+ 1920us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:43:16.744492 (+ 2272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:43:16.744498 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:43:16.744514 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:43:16.744888 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:43:16.745253 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:43:16.745257 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:43:16.745259 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:43:16.745432 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:43:16.745962 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:43:16.745968 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:43:16.746298 (+ 330us) client_negotiation.cc:769] Sending connection context
0927 12:43:16.747376 (+ 1078us) client_negotiation.cc:240] Negotiation successful
0927 12:43:16.747756 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1585,"thread_start_us":140,"threads_started":1}
I20260927 12:43:16.750438 12988 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40293
I20260927 12:43:16.750725 12988 heartbeater.cc:464] Registering TS with master...
I20260927 12:43:16.751303 12988 heartbeater.cc:511] Master 127.0.98.62:40293 requested a full tablet report, sending...
I20260927 12:43:16.752759 12638 ts_manager.cc:194] Registered new tserver with Master: 93287b8840454786a4944835236be293 (127.0.98.2:38825)
I20260927 12:43:16.753484 12638 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55975
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 12:43:16.792773 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:59175/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40293
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:43:16.948791 12993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:43:16.949203 12993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:43:16.949331 12993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:43:16.955715 12993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:43:16.955844 12993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:43:16.955977 12993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:43:16.963709 12993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:59175/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12993
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:43:16.965317 12993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:43:16.966782 12993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:43:16.980396 12999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:43:16.980906 13001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:43:16.981164 12998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:43:16.981839 12993 server_base.cc:1034] running on GCE node
I20260927 12:43:16.982400 12993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:43:16.983254 12993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:43:16.990682 12993 hybrid_clock.cc:648] HybridClock initialized: now 1790512996990641 us; error 60 us; skew 500 ppm
Sep 27 12:43:16 dist-test-slave-bnc1 krb5kdc[12162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512996, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:43:16.995190 12993 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 12:43:16.996697 12993 webserver.cc:460] Webserver started at http://127.0.98.3:39891/ using document root <none> and password file <none>
I20260927 12:43:16.997529 12993 fs_manager.cc:359] Metadata directory not provided
I20260927 12:43:16.997712 12993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:43:16.998052 12993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 12:43:16 dist-test-slave-bnc1 krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512996, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 12:43:17.026232 12993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-2/data/instance:
uuid: "9b07c5e0fa79429c87a5563d331d6d24"
format_stamp: "Formatted at 2026-09-27 12:43:17 on dist-test-slave-bnc1"
server_key: "874797fc4528d4a6d51cdb96815fca8e"
server_key_iv: "ac1b2d0c336a5e2b5c767eea39ec7695"
server_key_version: "kuduclusterkey@0"
I20260927 12:43:17.027076 12993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9b07c5e0fa79429c87a5563d331d6d24"
format_stamp: "Formatted at 2026-09-27 12:43:17 on dist-test-slave-bnc1"
server_key: "874797fc4528d4a6d51cdb96815fca8e"
server_key_iv: "ac1b2d0c336a5e2b5c767eea39ec7695"
server_key_version: "kuduclusterkey@0"
I20260927 12:43:17.032100 12993 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 12:43:17.055501 13008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:43:17.057297 12993 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.002s sys 0.000s
I20260927 12:43:17.057428 12993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-2/wal
uuid: "9b07c5e0fa79429c87a5563d331d6d24"
format_stamp: "Formatted at 2026-09-27 12:43:17 on dist-test-slave-bnc1"
server_key: "874797fc4528d4a6d51cdb96815fca8e"
server_key_iv: "ac1b2d0c336a5e2b5c767eea39ec7695"
server_key_version: "kuduclusterkey@0"
I20260927 12:43:17.057590 12993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:43:17.090338 12993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:43:17.094480 12993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:43:17.094744 12993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:43:17.095432 12993 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:43:17.096767 12993 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:43:17.096871 12993 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:43:17.096976 12993 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:43:17.097039 12993 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:43:17.112172 12993 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40563
I20260927 12:43:17.113816 12993 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:43:17.114967 13121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40563 every 8 connection(s)
I20260927 12:43:17.117844 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 12993
I20260927 12:43:17.117955 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-2/wal/instance
I20260927 12:43:17.118242 392 external_mini_cluster.cc:1428] Setting key ad6dbdd66f02fe8cff36f1bcab75e0a4
Sep 27 12:43:17 dist-test-slave-bnc1 krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512996, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:43:17.134476 12852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:17.119303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51773 (local address 127.0.98.62:40293)
0927 12:43:17.119424 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 12:43:17.119428 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:43:17.120388 (+ 960us) server_negotiation.cc:407] Connection header received
0927 12:43:17.121650 (+ 1262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:43:17.121653 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:43:17.121708 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:43:17.121759 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:43:17.123480 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:17.124296 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:17.126022 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:17.126582 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:17.126664 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:17.129975 (+ 3311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:43:17.129980 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:43:17.129982 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:43:17.130011 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:43:17.131820 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:17.132472 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:17.132476 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:17.132478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:17.132562 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:17.132983 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:17.132987 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:17.132989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:17.133234 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:43:17.133323 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 12:43:17.134161 (+ 838us) server_negotiation.cc:299] Negotiation successful
0927 12:43:17.134301 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 12:43:17.136555 13124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:17.117640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40293 (local address 127.0.98.3:51773)
0927 12:43:17.120241 (+ 2601us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:17.120295 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 12:43:17.121358 (+ 1063us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:43:17.122226 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:43:17.122235 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:43:17.122597 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:43:17.123310 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:17.123330 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:17.124457 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:17.124461 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:17.125877 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:17.125889 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:17.126893 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:17.126897 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:17.127112 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:17.127751 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:43:17.127773 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:43:17.129767 (+ 1994us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:43:17.131971 (+ 2204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:43:17.131979 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:43:17.131998 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:43:17.132357 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:43:17.132673 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:43:17.132676 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:43:17.132678 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:43:17.132872 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:43:17.133415 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:43:17.133423 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:43:17.133851 (+ 428us) client_negotiation.cc:769] Sending connection context
0927 12:43:17.134958 (+ 1107us) client_negotiation.cc:240] Negotiation successful
0927 12:43:17.135364 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2360,"thread_start_us":660,"threads_started":1}
I20260927 12:43:17.138644 13122 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40293
I20260927 12:43:17.138906 13122 heartbeater.cc:464] Registering TS with master...
I20260927 12:43:17.139525 13122 heartbeater.cc:511] Master 127.0.98.62:40293 requested a full tablet report, sending...
I20260927 12:43:17.140763 12638 ts_manager.cc:194] Registered new tserver with Master: 9b07c5e0fa79429c87a5563d331d6d24 (127.0.98.3:40563)
I20260927 12:43:17.141414 12638 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51773
Sep 27 12:43:17 dist-test-slave-bnc1 krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512996, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 12:43:17.144980 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 12:43:17.145656 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 12719
I20260927 12:43:17.155527 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 12856
I20260927 12:43:17.163715 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 12993
I20260927 12:43:17.173738 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 12606
I20260927 12:43:17.184657 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40293
--webserver_interface=127.0.98.62
--webserver_port=39805
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40293
--ranger_config_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:59175/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:59175/kms with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:43:17.310127 13125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:43:17.310359 13125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:43:17.310395 13125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:43:17.314174 13125 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:43:17.314217 13125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:43:17.314235 13125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:43:17.314251 13125 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:43:17.314265 13125 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:43:17.319355 13125 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:59175/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40293
--ranger_config_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=39805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13125
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:43:17.320297 13125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:43:17.321256 13125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:43:17.329638 13131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:43:17.329617 13130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:43:17.329687 13125 server_base.cc:1034] running on GCE node
W20260927 12:43:17.329810 13133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:43:17.330395 13125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:43:17.331570 13125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:43:17.332737 13125 hybrid_clock.cc:648] HybridClock initialized: now 1790512997332725 us; error 52 us; skew 500 ppm
Sep 27 12:43:17 dist-test-slave-bnc1 krb5kdc[12162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512997, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:43:17.335512 13125 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:43:17.336365 13125 webserver.cc:460] Webserver started at http://127.0.98.62:39805/ using document root <none> and password file <none>
I20260927 12:43:17.336890 13125 fs_manager.cc:359] Metadata directory not provided
I20260927 12:43:17.336977 13125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 12:43:17 dist-test-slave-bnc1 krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512997, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 12:43:17.402194 13125 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 12:43:17.404603 13141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:43:17.406150 13125 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:43:17.406275 13125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/master-0/wal
uuid: "38a6c6e565764580ac353d604b8b057b"
format_stamp: "Formatted at 2026-09-27 12:43:15 on dist-test-slave-bnc1"
server_key: "aa1d2f37ff60e9762a6174b0832ee63c"
server_key_iv: "95ebdd2906d5c6b8234b799ac832d006"
server_key_version: "kuduclusterkey@0"
I20260927 12:43:17.406857 13125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:43:17.416783 13125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:43:17.419402 13125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:43:17.419567 13125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:43:17.426777 13125 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40293
I20260927 12:43:17.426787 13193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40293 every 8 connection(s)
I20260927 12:43:17.427839 13125 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:43:17.432145 13194 sys_catalog.cc:263] Verifying existing consensus state
I20260927 12:43:17.432838 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 13125
I20260927 12:43:17.433338 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:59175/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:37985
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:40293
--webserver_port=41689
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:59175/kms with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 12:43:17.434361 13194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38a6c6e565764580ac353d604b8b057b: Bootstrap starting.
I20260927 12:43:17.450867 13194 log.cc:824] T 00000000000000000000000000000000 P 38a6c6e565764580ac353d604b8b057b: Log is configured to *not* fsync() on all Append() calls
I20260927 12:43:17.457151 13194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38a6c6e565764580ac353d604b8b057b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 12:43:17.457523 13194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38a6c6e565764580ac353d604b8b057b: Bootstrap complete.
I20260927 12:43:17.461675 13194 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38a6c6e565764580ac353d604b8b057b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38a6c6e565764580ac353d604b8b057b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40293 } }
I20260927 12:43:17.462047 13194 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38a6c6e565764580ac353d604b8b057b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38a6c6e565764580ac353d604b8b057b, State: Initialized, Role: FOLLOWER
I20260927 12:43:17.462514 13194 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38a6c6e565764580ac353d604b8b057b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38a6c6e565764580ac353d604b8b057b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40293 } }
I20260927 12:43:17.462630 13194 raft_consensus.cc:388] T 00000000000000000000000000000000 P 38a6c6e565764580ac353d604b8b057b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:43:17.462710 13194 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38a6c6e565764580ac353d604b8b057b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:43:17.462837 13194 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38a6c6e565764580ac353d604b8b057b [term 1 FOLLOWER]: Advancing to term 2
I20260927 12:43:17.464648 13194 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38a6c6e565764580ac353d604b8b057b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38a6c6e565764580ac353d604b8b057b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40293 } }
I20260927 12:43:17.464973 13194 leader_election.cc:302] T 00000000000000000000000000000000 P 38a6c6e565764580ac353d604b8b057b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38a6c6e565764580ac353d604b8b057b; no voters:
I20260927 12:43:17.465269 13194 leader_election.cc:288] T 00000000000000000000000000000000 P 38a6c6e565764580ac353d604b8b057b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 12:43:17.465405 13198 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38a6c6e565764580ac353d604b8b057b [term 2 FOLLOWER]: Leader election won for term 2
I20260927 12:43:17.465672 13198 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38a6c6e565764580ac353d604b8b057b [term 2 LEADER]: Becoming Leader. State: Replica: 38a6c6e565764580ac353d604b8b057b, State: Running, Role: LEADER
I20260927 12:43:17.466060 13198 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38a6c6e565764580ac353d604b8b057b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38a6c6e565764580ac353d604b8b057b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40293 } }
I20260927 12:43:17.466470 13194 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38a6c6e565764580ac353d604b8b057b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:43:17.467693 13200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38a6c6e565764580ac353d604b8b057b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38a6c6e565764580ac353d604b8b057b. Latest consensus state: current_term: 2 leader_uuid: "38a6c6e565764580ac353d604b8b057b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38a6c6e565764580ac353d604b8b057b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40293 } } }
I20260927 12:43:17.467820 13200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38a6c6e565764580ac353d604b8b057b [sys.catalog]: This master's current role is: LEADER
I20260927 12:43:17.467697 13199 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38a6c6e565764580ac353d604b8b057b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "38a6c6e565764580ac353d604b8b057b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38a6c6e565764580ac353d604b8b057b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40293 } } }
I20260927 12:43:17.468147 13199 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38a6c6e565764580ac353d604b8b057b [sys.catalog]: This master's current role is: LEADER
I20260927 12:43:17.468482 13207 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:43:17.468817 13194 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 12:43:17.475320 13207 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:43:17.476136 13207 catalog_manager.cc:1259] Loaded cluster ID: df8f3e8b28034e3a84b93c084799d0f2
I20260927 12:43:17.476230 13207 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:43:17.479141 13207 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:43:17.480036 13207 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 38a6c6e565764580ac353d604b8b057b: Loaded TSK: 0
I20260927 12:43:17.481491 13207 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 12:43:17.573719 13196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:43:17.574061 13196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:43:17.574127 13196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:43:17.580210 13196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:43:17.580355 13196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:43:17.580499 13196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:43:17.588158 13196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:59175/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:37985
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=41689
--tserver_master_addrs=127.0.98.62:40293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13196
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:43:17.589447 13196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:43:17.590740 13196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:43:17.603264 13231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:43:17.603571 13234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:43:17.603871 13232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:43:17.604846 13196 server_base.cc:1034] running on GCE node
I20260927 12:43:17.605353 13196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:43:17.606201 13196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:43:17.613670 13196 hybrid_clock.cc:648] HybridClock initialized: now 1790512997613626 us; error 44 us; skew 500 ppm
Sep 27 12:43:17 dist-test-slave-bnc1 krb5kdc[12162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512997, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:43:17.617717 13196 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 12:43:17.619019 13196 webserver.cc:460] Webserver started at http://127.0.98.1:41689/ using document root <none> and password file <none>
I20260927 12:43:17.619796 13196 fs_manager.cc:359] Metadata directory not provided
I20260927 12:43:17.619915 13196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 12:43:17 dist-test-slave-bnc1 krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512997, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 12:43:17.658113 13196 fs_manager.cc:709] Time spent opening directory manager: real 0.013s user 0.003s sys 0.000s
I20260927 12:43:17.661151 13242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:43:17.662853 13196 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 12:43:17.662989 13196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-0/wal
uuid: "3f555fa910234b098329094d7fee0dfb"
format_stamp: "Formatted at 2026-09-27 12:43:16 on dist-test-slave-bnc1"
server_key: "4e81ea75998c50963c31a30350b48219"
server_key_iv: "3e6ff185ca4806357b1536d03ad35103"
server_key_version: "kuduclusterkey@0"
I20260927 12:43:17.663736 13196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:43:17.682813 13196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:43:17.686777 13196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:43:17.686995 13196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:43:17.687703 13196 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:43:17.688799 13196 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:43:17.688862 13196 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:43:17.688922 13196 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:43:17.688953 13196 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:43:17.700582 13196 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37985
I20260927 12:43:17.700637 13355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37985 every 8 connection(s)
I20260927 12:43:17.701647 13196 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:43:17.701886 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 13196
I20260927 12:43:17.702443 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:59175/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:38825
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:40293
--webserver_port=39649
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:59175/kms with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 12:43:17 dist-test-slave-bnc1 krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512997, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:43:17.723891 13360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:17.704284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40293 (local address 127.0.98.1:39805)
0927 12:43:17.705321 (+ 1037us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:17.705361 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 12:43:17.706628 (+ 1267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:43:17.708518 (+ 1890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:43:17.708530 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:43:17.708902 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:43:17.709660 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:17.709685 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:17.711888 (+ 2203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:17.711892 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:17.713556 (+ 1664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:17.713569 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:17.714996 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:17.715002 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:17.715235 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:17.715944 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:43:17.715968 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:43:17.717935 (+ 1967us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:43:17.720471 (+ 2536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:43:17.720477 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:43:17.720490 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:43:17.720803 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:43:17.721177 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:43:17.721180 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:43:17.721182 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:43:17.721333 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:43:17.722057 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:43:17.722063 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:43:17.722353 (+ 290us) client_negotiation.cc:769] Sending connection context
0927 12:43:17.722528 (+ 175us) client_negotiation.cc:240] Negotiation successful
0927 12:43:17.722841 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":876,"thread_start_us":110,"threads_started":1}
I20260927 12:43:17.725550 13359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:17.704134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39805 (local address 127.0.98.62:40293)
0927 12:43:17.704674 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0927 12:43:17.704703 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0927 12:43:17.705496 (+ 793us) server_negotiation.cc:407] Connection header received
0927 12:43:17.707223 (+ 1727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:43:17.707241 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:43:17.707646 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:43:17.707977 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:43:17.709953 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:17.711722 (+ 1769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:17.713724 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:17.714554 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:17.714738 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:17.718148 (+ 3410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:43:17.718154 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:43:17.718169 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:43:17.718198 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:43:17.720344 (+ 2146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:17.720930 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:17.720935 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:17.720940 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:17.721064 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:17.721429 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:17.721433 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:17.721434 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:17.721886 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:43:17.722004 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 12:43:17.724422 (+ 2418us) server_negotiation.cc:299] Negotiation successful
0927 12:43:17.724694 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":124,"threads_started":1}
I20260927 12:43:17.728868 13356 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40293
I20260927 12:43:17.729095 13356 heartbeater.cc:464] Registering TS with master...
I20260927 12:43:17.729646 13356 heartbeater.cc:511] Master 127.0.98.62:40293 requested a full tablet report, sending...
W20260927 12:43:17.731833 13356 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40293 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 12:43:17.731946 13356 heartbeater.cc:464] Registering TS with master...
I20260927 12:43:17.732129 13356 heartbeater.cc:511] Master 127.0.98.62:40293 requested a full tablet report, sending...
I20260927 12:43:17.733032 13356 heartbeater.cc:464] Registering TS with master...
I20260927 12:43:17.733191 13356 heartbeater.cc:511] Master 127.0.98.62:40293 requested a full tablet report, sending...
W20260927 12:43:17.733968 13356 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 12:43:17.841495 13358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:43:17.841928 13358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:43:17.842016 13358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:43:17.846534 13358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:43:17.846626 13358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:43:17.846743 13358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:43:17.852290 13358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:59175/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:38825
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=39649
--tserver_master_addrs=127.0.98.62:40293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13358
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:43:17.853437 13358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:43:17.854486 13358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:43:17.864491 13368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:43:17.864462 13366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:43:17.864959 13365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:43:17.865849 13358 server_base.cc:1034] running on GCE node
I20260927 12:43:17.866266 13358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:43:17.866968 13358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:43:17.868147 13358 hybrid_clock.cc:648] HybridClock initialized: now 1790512997868109 us; error 43 us; skew 500 ppm
Sep 27 12:43:17 dist-test-slave-bnc1 krb5kdc[12162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512997, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:43:17.871212 13358 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 12:43:17.872099 13358 webserver.cc:460] Webserver started at http://127.0.98.2:39649/ using document root <none> and password file <none>
I20260927 12:43:17.872617 13358 fs_manager.cc:359] Metadata directory not provided
I20260927 12:43:17.872722 13358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 12:43:17 dist-test-slave-bnc1 krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512997, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 12:43:17.908217 13358 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 12:43:17.912386 13376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:43:17.913978 13358 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260927 12:43:17.914183 13358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-1/wal
uuid: "93287b8840454786a4944835236be293"
format_stamp: "Formatted at 2026-09-27 12:43:16 on dist-test-slave-bnc1"
server_key: "094ce18f68ed1f915dbbbe5afb0f5124"
server_key_iv: "1f716075cf969b96fdead3a6fe962892"
server_key_version: "kuduclusterkey@0"
I20260927 12:43:17.915010 13358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:43:17.928128 13358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:43:17.931732 13358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:43:17.932078 13358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:43:17.932835 13358 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:43:17.933804 13358 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:43:17.933878 13358 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:43:17.933948 13358 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:43:17.934013 13358 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:43:17.946177 13358 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38825
I20260927 12:43:17.947273 13358 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:43:17.952445 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 13358
I20260927 12:43:17.952984 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:59175/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:40563
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:40293
--webserver_port=39891
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:59175/kms with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 12:43:17 dist-test-slave-bnc1 krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512997, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:43:17.963792 13489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38825 every 8 connection(s)
I20260927 12:43:17.966032 13492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:17.949560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40293 (local address 127.0.98.2:58051)
0927 12:43:17.950157 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:17.950212 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 12:43:17.951109 (+ 897us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:43:17.951703 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:43:17.951710 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:43:17.952096 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:43:17.952906 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:17.952930 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:17.954504 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:17.954509 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:17.955542 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:17.955552 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:17.956393 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:17.956399 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:17.956631 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:17.957346 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:43:17.957373 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:43:17.959303 (+ 1930us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:43:17.961840 (+ 2537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:43:17.961849 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:43:17.961868 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:43:17.962301 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:43:17.962698 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:43:17.962704 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:43:17.962710 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:43:17.962930 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:43:17.963594 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:43:17.963604 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:43:17.964118 (+ 514us) client_negotiation.cc:769] Sending connection context
0927 12:43:17.964371 (+ 253us) client_negotiation.cc:240] Negotiation successful
0927 12:43:17.964769 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":165,"threads_started":1}
I20260927 12:43:17.967586 13359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:17.949731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58051 (local address 127.0.98.62:40293)
0927 12:43:17.949895 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 12:43:17.949900 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:43:17.950319 (+ 419us) server_negotiation.cc:407] Connection header received
0927 12:43:17.951336 (+ 1017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:43:17.951340 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:43:17.951403 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:43:17.951518 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:43:17.953409 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:17.954296 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:17.955676 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:17.956085 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:17.956203 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:17.959542 (+ 3339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:43:17.959549 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:43:17.959555 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:43:17.959596 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:43:17.961672 (+ 2076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:17.962452 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:17.962458 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:17.962460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:17.962565 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:17.963100 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:17.963106 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:17.963109 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:17.963402 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:43:17.963522 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 12:43:17.966712 (+ 3190us) server_negotiation.cc:299] Negotiation successful
0927 12:43:17.966879 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 12:43:17.969276 13490 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40293
I20260927 12:43:17.969554 13490 heartbeater.cc:464] Registering TS with master...
I20260927 12:43:17.970112 13490 heartbeater.cc:511] Master 127.0.98.62:40293 requested a full tablet report, sending...
W20260927 12:43:17.971813 13490 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40293 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 12:43:17.971956 13490 heartbeater.cc:464] Registering TS with master...
I20260927 12:43:17.972139 13490 heartbeater.cc:511] Master 127.0.98.62:40293 requested a full tablet report, sending...
I20260927 12:43:17.972993 13490 heartbeater.cc:464] Registering TS with master...
I20260927 12:43:17.973198 13490 heartbeater.cc:511] Master 127.0.98.62:40293 requested a full tablet report, sending...
W20260927 12:43:17.974073 13490 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 12:43:18.097604 13493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:43:18.097922 13493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:43:18.098016 13493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:43:18.102371 13493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:43:18.102447 13493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:43:18.102532 13493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:43:18.109851 13493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:59175/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:40563
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=39891
--tserver_master_addrs=127.0.98.62:40293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:43:18.111200 13493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:43:18.112381 13493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:43:18.125190 13500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:43:18.125159 13499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:43:18.125172 13502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:43:18.125778 13493 server_base.cc:1034] running on GCE node
I20260927 12:43:18.126494 13493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:43:18.127311 13493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:43:18.128530 13493 hybrid_clock.cc:648] HybridClock initialized: now 1790512998128495 us; error 63 us; skew 500 ppm
Sep 27 12:43:18 dist-test-slave-bnc1 krb5kdc[12162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512998, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:43:18.132082 13493 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 12:43:18.133253 13493 webserver.cc:460] Webserver started at http://127.0.98.3:39891/ using document root <none> and password file <none>
I20260927 12:43:18.134037 13493 fs_manager.cc:359] Metadata directory not provided
I20260927 12:43:18.134151 13493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 12:43:18 dist-test-slave-bnc1 krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512998, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 12:43:18.153314 13493 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260927 12:43:18.156654 13510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:43:18.158411 13493 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260927 12:43:18.158577 13493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-2/wal
uuid: "9b07c5e0fa79429c87a5563d331d6d24"
format_stamp: "Formatted at 2026-09-27 12:43:17 on dist-test-slave-bnc1"
server_key: "874797fc4528d4a6d51cdb96815fca8e"
server_key_iv: "ac1b2d0c336a5e2b5c767eea39ec7695"
server_key_version: "kuduclusterkey@0"
I20260927 12:43:18.159343 13493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:43:18.180471 13493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:43:18.184295 13493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:43:18.184526 13493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:43:18.185377 13493 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:43:18.186587 13493 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:43:18.186662 13493 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:43:18.186738 13493 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:43:18.186780 13493 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:43:18.198334 13493 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40563
I20260927 12:43:18.199129 13623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40563 every 8 connection(s)
I20260927 12:43:18.199752 13493 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:43:18.202571 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 13493
Sep 27 12:43:18 dist-test-slave-bnc1 krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512998, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:43:18.218446 13627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:18.206080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44851 (local address 127.0.98.62:40293)
0927 12:43:18.206332 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0927 12:43:18.206336 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:43:18.207245 (+ 909us) server_negotiation.cc:407] Connection header received
0927 12:43:18.208246 (+ 1001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:43:18.208249 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:43:18.208302 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:43:18.208385 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:43:18.209752 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:18.210552 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:18.211920 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:18.212175 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:18.212284 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:18.214771 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:43:18.214774 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:43:18.214777 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:43:18.214802 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:43:18.216332 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:18.216853 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:18.216857 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:18.216859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:18.216941 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:18.217290 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:18.217293 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:18.217295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:18.217473 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:43:18.217602 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 12:43:18.218136 (+ 534us) server_negotiation.cc:299] Negotiation successful
0927 12:43:18.218256 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":98,"threads_started":1}
I20260927 12:43:18.219477 13628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:18.206641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40293 (local address 127.0.98.3:44851)
0927 12:43:18.207102 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:18.207156 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 12:43:18.208037 (+ 881us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:43:18.208565 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:43:18.208581 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:43:18.208862 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:43:18.209556 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:18.209627 (+ 71us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:18.210679 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:18.210682 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:18.211805 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:18.211815 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:18.212326 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:18.212330 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:18.212511 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:18.213118 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:43:18.213135 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:43:18.214608 (+ 1473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:43:18.216466 (+ 1858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:43:18.216471 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:43:18.216487 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:43:18.216750 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:43:18.217036 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:43:18.217039 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:43:18.217041 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:43:18.217189 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:43:18.217617 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:43:18.217624 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:43:18.217889 (+ 265us) client_negotiation.cc:769] Sending connection context
0927 12:43:18.218077 (+ 188us) client_negotiation.cc:240] Negotiation successful
0927 12:43:18.218391 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":138,"threads_started":1}
I20260927 12:43:18.220767 13624 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40293
I20260927 12:43:18.221033 13624 heartbeater.cc:464] Registering TS with master...
I20260927 12:43:18.221803 13624 heartbeater.cc:511] Master 127.0.98.62:40293 requested a full tablet report, sending...
W20260927 12:43:18.223490 13624 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40293 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 12:43:18.223963 13624 heartbeater.cc:464] Registering TS with master...
I20260927 12:43:18.224159 13624 heartbeater.cc:511] Master 127.0.98.62:40293 requested a full tablet report, sending...
I20260927 12:43:18.225323 13624 heartbeater.cc:464] Registering TS with master...
I20260927 12:43:18.225507 13624 heartbeater.cc:511] Master 127.0.98.62:40293 requested a full tablet report, sending...
W20260927 12:43:18.226310 13624 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 12:43:18.227393 13359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:18.203302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41716 (local address 127.0.98.62:40293)
0927 12:43:18.203442 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 12:43:18.203446 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:43:18.203720 (+ 274us) server_negotiation.cc:407] Connection header received
0927 12:43:18.203917 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:43:18.203921 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:43:18.203978 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:43:18.204068 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:43:18.204921 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:18.205646 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:18.208959 (+ 3313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:18.209281 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:18.209389 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:18.223234 (+ 13845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:43:18.223237 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:43:18.223240 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:43:18.223266 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:43:18.225128 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:18.225847 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:18.225850 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:18.225852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:18.225939 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:18.226525 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:18.226529 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:18.226531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:18.226741 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:43:18.226973 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0927 12:43:18.227110 (+ 137us) server_negotiation.cc:299] Negotiation successful
0927 12:43:18.227233 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Sep 27 12:43:18 dist-test-slave-bnc1 krb5kdc[12162](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790512998, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:43:18.734169 13356 heartbeater.cc:464] Registering TS with master...
I20260927 12:43:18.734470 13356 heartbeater.cc:511] Master 127.0.98.62:40293 requested a full tablet report, sending...
I20260927 12:43:18.735615 13158 ts_manager.cc:194] Registered new tserver with Master: 3f555fa910234b098329094d7fee0dfb (127.0.98.1:37985)
I20260927 12:43:18.737336 13158 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39805
I20260927 12:43:18.974552 13490 heartbeater.cc:464] Registering TS with master...
I20260927 12:43:18.974886 13490 heartbeater.cc:511] Master 127.0.98.62:40293 requested a full tablet report, sending...
I20260927 12:43:18.975728 13158 ts_manager.cc:194] Registered new tserver with Master: 93287b8840454786a4944835236be293 (127.0.98.2:38825)
I20260927 12:43:18.979228 13158 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58051
Sep 27 12:43:18 dist-test-slave-bnc1 krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512998, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 12:43:19.226541 13624 heartbeater.cc:464] Registering TS with master...
I20260927 12:43:19.226886 13624 heartbeater.cc:511] Master 127.0.98.62:40293 requested a full tablet report, sending...
I20260927 12:43:19.227581 13158 ts_manager.cc:194] Registered new tserver with Master: 9b07c5e0fa79429c87a5563d331d6d24 (127.0.98.3:40563)
I20260927 12:43:19.228430 13158 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44851
I20260927 12:43:19.236537 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 12:43:19.252373 13647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:19.238247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41730 (local address 127.0.98.62:40293)
0927 12:43:19.238724 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0927 12:43:19.238729 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:43:19.242052 (+ 3323us) server_negotiation.cc:407] Connection header received
0927 12:43:19.242101 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:43:19.242104 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:43:19.242156 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:43:19.242250 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:43:19.243057 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:19.243866 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:19.245060 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:19.245396 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:19.245488 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:19.248093 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:43:19.248097 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:43:19.248100 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:43:19.248127 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:43:19.250142 (+ 2015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:19.250624 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:19.250628 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:19.250630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:19.250732 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:19.251230 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:19.251235 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:19.251238 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:19.251487 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:43:19.251637 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 12:43:19.251880 (+ 243us) server_negotiation.cc:299] Negotiation successful
0927 12:43:19.252011 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":79,"threads_started":1}
Sep 27 12:43:19 dist-test-slave-bnc1 krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790512998, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260927 12:43:19.257189 13158 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41730:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 12:43:19.263370 13158 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:43:19.289517 13656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:19.280011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38825 (local address 127.0.0.1:34696)
0927 12:43:19.282918 (+ 2907us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:19.282971 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 12:43:19.283105 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:43:19.284240 (+ 1135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:43:19.284246 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:43:19.284269 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:43:19.284401 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:19.284411 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:19.286189 (+ 1778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:19.286193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:19.287533 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:19.287541 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:19.288920 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:19.288924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:19.289042 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:19.289100 (+ 58us) client_negotiation.cc:769] Sending connection context
0927 12:43:19.289224 (+ 124us) client_negotiation.cc:240] Negotiation successful
0927 12:43:19.289356 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2725,"thread_start_us":123,"threads_started":1}
I20260927 12:43:19.296681 13654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:19.280853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34696 (local address 127.0.98.2:38825)
0927 12:43:19.283563 (+ 2710us) server_negotiation.cc:206] Beginning negotiation
0927 12:43:19.283569 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:43:19.283606 (+ 37us) server_negotiation.cc:407] Connection header received
0927 12:43:19.283686 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:43:19.283691 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:43:19.283906 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:43:19.284052 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:43:19.284833 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:19.286045 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:19.287685 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:19.288562 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:19.288689 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:19.288850 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 12:43:19.296368 (+ 7518us) server_negotiation.cc:299] Negotiation successful
0927 12:43:19.296498 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2612,"thread_start_us":114,"threads_started":1}
I20260927 12:43:19.299495 13656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:19.281506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40563 (local address 127.0.0.1:44926)
0927 12:43:19.290218 (+ 8712us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:19.290232 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:43:19.290395 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:43:19.291177 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:43:19.291180 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:43:19.291197 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:43:19.291274 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:19.291280 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:19.292708 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:19.292712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:19.293855 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:19.293866 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:19.299114 (+ 5248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:19.299118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:19.299204 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:19.299229 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 12:43:19.299301 (+ 72us) client_negotiation.cc:240] Negotiation successful
0927 12:43:19.299368 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8652,"thread_start_us":114,"threads_started":1}
I20260927 12:43:19.300040 13657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:19.281922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44926 (local address 127.0.98.3:40563)
0927 12:43:19.283384 (+ 1462us) server_negotiation.cc:206] Beginning negotiation
0927 12:43:19.283391 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 12:43:19.290318 (+ 6927us) server_negotiation.cc:407] Connection header received
0927 12:43:19.290564 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:43:19.290570 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:43:19.290833 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:43:19.290965 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:43:19.291400 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:19.292554 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:19.294000 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:19.294845 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:19.294946 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:19.295097 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 12:43:19.299806 (+ 4709us) server_negotiation.cc:299] Negotiation successful
0927 12:43:19.299910 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1357,"thread_start_us":118,"threads_started":1}
I20260927 12:43:19.300261 13424 tablet_service.cc:1467] Processing CreateTablet for tablet dbc9174d572f4ad896d35c434728dbfd (DEFAULT_TABLE table=test-table [id=8aa7dfc74d7f4942b99914748ccd16d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:43:19.301461 13424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbc9174d572f4ad896d35c434728dbfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:43:19.303164 13558 tablet_service.cc:1467] Processing CreateTablet for tablet dbc9174d572f4ad896d35c434728dbfd (DEFAULT_TABLE table=test-table [id=8aa7dfc74d7f4942b99914748ccd16d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:43:19.304193 13558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbc9174d572f4ad896d35c434728dbfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:43:19.313508 13659 tablet_bootstrap.cc:492] T dbc9174d572f4ad896d35c434728dbfd P 93287b8840454786a4944835236be293: Bootstrap starting.
I20260927 12:43:19.316970 13659 tablet_bootstrap.cc:654] T dbc9174d572f4ad896d35c434728dbfd P 93287b8840454786a4944835236be293: Neither blocks nor log segments found. Creating new log.
I20260927 12:43:19.317967 13659 log.cc:824] T dbc9174d572f4ad896d35c434728dbfd P 93287b8840454786a4944835236be293: Log is configured to *not* fsync() on all Append() calls
I20260927 12:43:19.318737 13653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:19.280353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37985 (local address 127.0.0.1:35410)
0927 12:43:19.284629 (+ 4276us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:19.284644 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:43:19.284721 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:43:19.296098 (+ 11377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:43:19.296101 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:43:19.296119 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:43:19.296204 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:19.296211 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:19.310948 (+ 14737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:19.310953 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:19.312165 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:19.312177 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:19.318245 (+ 6068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:19.318251 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:19.318363 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:19.318394 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 12:43:19.318456 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 12:43:19.318550 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4204,"thread_start_us":90,"threads_started":1}
I20260927 12:43:19.318733 13660 tablet_bootstrap.cc:492] T dbc9174d572f4ad896d35c434728dbfd P 9b07c5e0fa79429c87a5563d331d6d24: Bootstrap starting.
I20260927 12:43:19.319739 13655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:19.281000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35410 (local address 127.0.98.1:37985)
0927 12:43:19.295401 (+ 14401us) server_negotiation.cc:206] Beginning negotiation
0927 12:43:19.295411 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 12:43:19.295441 (+ 30us) server_negotiation.cc:407] Connection header received
0927 12:43:19.295523 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:43:19.295531 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:43:19.295766 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:43:19.295930 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:43:19.297878 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:19.298927 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:19.315013 (+ 16086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:19.315978 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:19.319225 (+ 3247us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:19.319406 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0927 12:43:19.319463 (+ 57us) server_negotiation.cc:299] Negotiation successful
0927 12:43:19.319585 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14278,"thread_start_us":103,"threads_started":1}
I20260927 12:43:19.320886 13660 tablet_bootstrap.cc:654] T dbc9174d572f4ad896d35c434728dbfd P 9b07c5e0fa79429c87a5563d331d6d24: Neither blocks nor log segments found. Creating new log.
I20260927 12:43:19.322216 13660 log.cc:824] T dbc9174d572f4ad896d35c434728dbfd P 9b07c5e0fa79429c87a5563d331d6d24: Log is configured to *not* fsync() on all Append() calls
I20260927 12:43:19.323169 13290 tablet_service.cc:1467] Processing CreateTablet for tablet dbc9174d572f4ad896d35c434728dbfd (DEFAULT_TABLE table=test-table [id=8aa7dfc74d7f4942b99914748ccd16d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:43:19.324270 13290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbc9174d572f4ad896d35c434728dbfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:43:19.324404 13660 tablet_bootstrap.cc:492] T dbc9174d572f4ad896d35c434728dbfd P 9b07c5e0fa79429c87a5563d331d6d24: No bootstrap required, opened a new log
I20260927 12:43:19.324410 13659 tablet_bootstrap.cc:492] T dbc9174d572f4ad896d35c434728dbfd P 93287b8840454786a4944835236be293: No bootstrap required, opened a new log
I20260927 12:43:19.324585 13659 ts_tablet_manager.cc:1397] T dbc9174d572f4ad896d35c434728dbfd P 93287b8840454786a4944835236be293: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20260927 12:43:19.324579 13660 ts_tablet_manager.cc:1397] T dbc9174d572f4ad896d35c434728dbfd P 9b07c5e0fa79429c87a5563d331d6d24: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260927 12:43:19.329185 13660 raft_consensus.cc:348] T dbc9174d572f4ad896d35c434728dbfd P 9b07c5e0fa79429c87a5563d331d6d24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93287b8840454786a4944835236be293" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38825 } } peers { permanent_uuid: "3f555fa910234b098329094d7fee0dfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37985 } } peers { permanent_uuid: "9b07c5e0fa79429c87a5563d331d6d24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40563 } }
I20260927 12:43:19.329558 13660 raft_consensus.cc:374] T dbc9174d572f4ad896d35c434728dbfd P 9b07c5e0fa79429c87a5563d331d6d24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:43:19.329696 13660 raft_consensus.cc:729] T dbc9174d572f4ad896d35c434728dbfd P 9b07c5e0fa79429c87a5563d331d6d24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b07c5e0fa79429c87a5563d331d6d24, State: Initialized, Role: FOLLOWER
I20260927 12:43:19.329875 13659 raft_consensus.cc:348] T dbc9174d572f4ad896d35c434728dbfd P 93287b8840454786a4944835236be293 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93287b8840454786a4944835236be293" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38825 } } peers { permanent_uuid: "3f555fa910234b098329094d7fee0dfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37985 } } peers { permanent_uuid: "9b07c5e0fa79429c87a5563d331d6d24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40563 } }
I20260927 12:43:19.330179 13659 raft_consensus.cc:374] T dbc9174d572f4ad896d35c434728dbfd P 93287b8840454786a4944835236be293 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:43:19.330246 13659 raft_consensus.cc:729] T dbc9174d572f4ad896d35c434728dbfd P 93287b8840454786a4944835236be293 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93287b8840454786a4944835236be293, State: Initialized, Role: FOLLOWER
I20260927 12:43:19.330374 13660 consensus_queue.cc:260] T dbc9174d572f4ad896d35c434728dbfd P 9b07c5e0fa79429c87a5563d331d6d24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93287b8840454786a4944835236be293" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38825 } } peers { permanent_uuid: "3f555fa910234b098329094d7fee0dfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37985 } } peers { permanent_uuid: "9b07c5e0fa79429c87a5563d331d6d24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40563 } }
I20260927 12:43:19.331521 13660 ts_tablet_manager.cc:1428] T dbc9174d572f4ad896d35c434728dbfd P 9b07c5e0fa79429c87a5563d331d6d24: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260927 12:43:19.331840 13659 consensus_queue.cc:260] T dbc9174d572f4ad896d35c434728dbfd P 93287b8840454786a4944835236be293 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93287b8840454786a4944835236be293" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38825 } } peers { permanent_uuid: "3f555fa910234b098329094d7fee0dfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37985 } } peers { permanent_uuid: "9b07c5e0fa79429c87a5563d331d6d24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40563 } }
I20260927 12:43:19.332002 13624 heartbeater.cc:503] Master 127.0.98.62:40293 was elected leader, sending a full tablet report...
I20260927 12:43:19.335932 13665 tablet_bootstrap.cc:492] T dbc9174d572f4ad896d35c434728dbfd P 3f555fa910234b098329094d7fee0dfb: Bootstrap starting.
I20260927 12:43:19.336433 13659 ts_tablet_manager.cc:1428] T dbc9174d572f4ad896d35c434728dbfd P 93287b8840454786a4944835236be293: Time spent starting tablet: real 0.012s user 0.008s sys 0.000s
I20260927 12:43:19.336828 13490 heartbeater.cc:503] Master 127.0.98.62:40293 was elected leader, sending a full tablet report...
I20260927 12:43:19.338761 13665 tablet_bootstrap.cc:654] T dbc9174d572f4ad896d35c434728dbfd P 3f555fa910234b098329094d7fee0dfb: Neither blocks nor log segments found. Creating new log.
I20260927 12:43:19.340921 13665 log.cc:824] T dbc9174d572f4ad896d35c434728dbfd P 3f555fa910234b098329094d7fee0dfb: Log is configured to *not* fsync() on all Append() calls
I20260927 12:43:19.344555 13665 tablet_bootstrap.cc:492] T dbc9174d572f4ad896d35c434728dbfd P 3f555fa910234b098329094d7fee0dfb: No bootstrap required, opened a new log
I20260927 12:43:19.344772 13665 ts_tablet_manager.cc:1397] T dbc9174d572f4ad896d35c434728dbfd P 3f555fa910234b098329094d7fee0dfb: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260927 12:43:19.349741 13665 raft_consensus.cc:348] T dbc9174d572f4ad896d35c434728dbfd P 3f555fa910234b098329094d7fee0dfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93287b8840454786a4944835236be293" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38825 } } peers { permanent_uuid: "3f555fa910234b098329094d7fee0dfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37985 } } peers { permanent_uuid: "9b07c5e0fa79429c87a5563d331d6d24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40563 } }
I20260927 12:43:19.350042 13665 raft_consensus.cc:374] T dbc9174d572f4ad896d35c434728dbfd P 3f555fa910234b098329094d7fee0dfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:43:19.350118 13665 raft_consensus.cc:729] T dbc9174d572f4ad896d35c434728dbfd P 3f555fa910234b098329094d7fee0dfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f555fa910234b098329094d7fee0dfb, State: Initialized, Role: FOLLOWER
I20260927 12:43:19.350663 13665 consensus_queue.cc:260] T dbc9174d572f4ad896d35c434728dbfd P 3f555fa910234b098329094d7fee0dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93287b8840454786a4944835236be293" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38825 } } peers { permanent_uuid: "3f555fa910234b098329094d7fee0dfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37985 } } peers { permanent_uuid: "9b07c5e0fa79429c87a5563d331d6d24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40563 } }
I20260927 12:43:19.353152 13665 ts_tablet_manager.cc:1428] T dbc9174d572f4ad896d35c434728dbfd P 3f555fa910234b098329094d7fee0dfb: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260927 12:43:19.353531 13356 heartbeater.cc:503] Master 127.0.98.62:40293 was elected leader, sending a full tablet report...
I20260927 12:43:19.423022 13667 raft_consensus.cc:482] T dbc9174d572f4ad896d35c434728dbfd P 3f555fa910234b098329094d7fee0dfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:43:19.423285 13667 raft_consensus.cc:504] T dbc9174d572f4ad896d35c434728dbfd P 3f555fa910234b098329094d7fee0dfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93287b8840454786a4944835236be293" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38825 } } peers { permanent_uuid: "3f555fa910234b098329094d7fee0dfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37985 } } peers { permanent_uuid: "9b07c5e0fa79429c87a5563d331d6d24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40563 } }
I20260927 12:43:19.424595 13667 leader_election.cc:288] T dbc9174d572f4ad896d35c434728dbfd P 3f555fa910234b098329094d7fee0dfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93287b8840454786a4944835236be293 (127.0.98.2:38825), 9b07c5e0fa79429c87a5563d331d6d24 (127.0.98.3:40563)
I20260927 12:43:19.430565 13654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:19.425779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43391 (local address 127.0.98.2:38825)
0927 12:43:19.425894 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 12:43:19.425899 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:43:19.426261 (+ 362us) server_negotiation.cc:407] Connection header received
0927 12:43:19.426463 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:43:19.426467 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:43:19.426515 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:43:19.426561 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:43:19.427011 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:19.427803 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:19.429266 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:19.429894 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:19.429987 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:19.430060 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 12:43:19.430366 (+ 306us) server_negotiation.cc:299] Negotiation successful
0927 12:43:19.430441 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260927 12:43:19.430917 13669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:19.424858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38825 (local address 127.0.98.1:43391)
0927 12:43:19.426160 (+ 1302us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:19.426185 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 12:43:19.426341 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:43:19.426695 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:43:19.426698 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:43:19.426717 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:43:19.426869 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:19.426877 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:19.427955 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:19.427959 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:19.429111 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:19.429120 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:19.430114 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:19.430120 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:19.430230 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:19.430253 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 12:43:19.430703 (+ 450us) client_negotiation.cc:240] Negotiation successful
0927 12:43:19.430773 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1237,"thread_start_us":114,"threads_started":1}
I20260927 12:43:19.431852 13444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbc9174d572f4ad896d35c434728dbfd" candidate_uuid: "3f555fa910234b098329094d7fee0dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93287b8840454786a4944835236be293" is_pre_election: true
I20260927 12:43:19.432197 13444 raft_consensus.cc:2454] T dbc9174d572f4ad896d35c434728dbfd P 93287b8840454786a4944835236be293 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f555fa910234b098329094d7fee0dfb in term 0.
I20260927 12:43:19.432683 13245 leader_election.cc:302] T dbc9174d572f4ad896d35c434728dbfd P 3f555fa910234b098329094d7fee0dfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f555fa910234b098329094d7fee0dfb, 93287b8840454786a4944835236be293; no voters:
I20260927 12:43:19.432958 13667 raft_consensus.cc:2781] T dbc9174d572f4ad896d35c434728dbfd P 3f555fa910234b098329094d7fee0dfb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:43:19.433071 13667 raft_consensus.cc:482] T dbc9174d572f4ad896d35c434728dbfd P 3f555fa910234b098329094d7fee0dfb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:43:19.433123 13667 raft_consensus.cc:3037] T dbc9174d572f4ad896d35c434728dbfd P 3f555fa910234b098329094d7fee0dfb [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:43:19.433866 13657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:19.425910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58223 (local address 127.0.98.3:40563)
0927 12:43:19.426014 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0927 12:43:19.426018 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:43:19.429508 (+ 3490us) server_negotiation.cc:407] Connection header received
0927 12:43:19.429706 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:43:19.429709 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:43:19.429759 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:43:19.429806 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:43:19.430171 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:19.430875 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:19.432644 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:19.433262 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:19.433326 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:19.433372 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 12:43:19.433694 (+ 322us) server_negotiation.cc:299] Negotiation successful
0927 12:43:19.433762 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260927 12:43:19.433863 13668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:19.425346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40563 (local address 127.0.98.1:58223)
0927 12:43:19.429427 (+ 4081us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:19.429440 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:43:19.429591 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:43:19.429933 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:43:19.429939 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:43:19.429960 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:43:19.430051 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:19.430056 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:19.431456 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:19.431459 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:19.432522 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:19.432531 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:19.433445 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:19.433447 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:19.433519 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:19.433533 (+ 14us) client_negotiation.cc:769] Sending connection context
0927 12:43:19.433670 (+ 137us) client_negotiation.cc:240] Negotiation successful
0927 12:43:19.433736 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4019,"thread_start_us":88,"threads_started":1}
I20260927 12:43:19.434437 13578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbc9174d572f4ad896d35c434728dbfd" candidate_uuid: "3f555fa910234b098329094d7fee0dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b07c5e0fa79429c87a5563d331d6d24" is_pre_election: true
I20260927 12:43:19.434464 13667 raft_consensus.cc:504] T dbc9174d572f4ad896d35c434728dbfd P 3f555fa910234b098329094d7fee0dfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93287b8840454786a4944835236be293" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38825 } } peers { permanent_uuid: "3f555fa910234b098329094d7fee0dfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37985 } } peers { permanent_uuid: "9b07c5e0fa79429c87a5563d331d6d24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40563 } }
I20260927 12:43:19.434713 13578 raft_consensus.cc:2454] T dbc9174d572f4ad896d35c434728dbfd P 9b07c5e0fa79429c87a5563d331d6d24 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f555fa910234b098329094d7fee0dfb in term 0.
I20260927 12:43:19.434885 13667 leader_election.cc:288] T dbc9174d572f4ad896d35c434728dbfd P 3f555fa910234b098329094d7fee0dfb [CANDIDATE]: Term 1 election: Requested vote from peers 93287b8840454786a4944835236be293 (127.0.98.2:38825), 9b07c5e0fa79429c87a5563d331d6d24 (127.0.98.3:40563)
I20260927 12:43:19.435194 13444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbc9174d572f4ad896d35c434728dbfd" candidate_uuid: "3f555fa910234b098329094d7fee0dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93287b8840454786a4944835236be293"
I20260927 12:43:19.435206 13578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbc9174d572f4ad896d35c434728dbfd" candidate_uuid: "3f555fa910234b098329094d7fee0dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b07c5e0fa79429c87a5563d331d6d24"
I20260927 12:43:19.435314 13444 raft_consensus.cc:3037] T dbc9174d572f4ad896d35c434728dbfd P 93287b8840454786a4944835236be293 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:43:19.435333 13578 raft_consensus.cc:3037] T dbc9174d572f4ad896d35c434728dbfd P 9b07c5e0fa79429c87a5563d331d6d24 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:43:19.436398 13578 raft_consensus.cc:2454] T dbc9174d572f4ad896d35c434728dbfd P 9b07c5e0fa79429c87a5563d331d6d24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f555fa910234b098329094d7fee0dfb in term 1.
I20260927 12:43:19.436547 13444 raft_consensus.cc:2454] T dbc9174d572f4ad896d35c434728dbfd P 93287b8840454786a4944835236be293 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f555fa910234b098329094d7fee0dfb in term 1.
I20260927 12:43:19.436925 13245 leader_election.cc:302] T dbc9174d572f4ad896d35c434728dbfd P 3f555fa910234b098329094d7fee0dfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f555fa910234b098329094d7fee0dfb, 9b07c5e0fa79429c87a5563d331d6d24; no voters:
I20260927 12:43:19.437256 13667 raft_consensus.cc:2781] T dbc9174d572f4ad896d35c434728dbfd P 3f555fa910234b098329094d7fee0dfb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:43:19.437472 13667 raft_consensus.cc:686] T dbc9174d572f4ad896d35c434728dbfd P 3f555fa910234b098329094d7fee0dfb [term 1 LEADER]: Becoming Leader. State: Replica: 3f555fa910234b098329094d7fee0dfb, State: Running, Role: LEADER
I20260927 12:43:19.437825 13667 consensus_queue.cc:237] T dbc9174d572f4ad896d35c434728dbfd P 3f555fa910234b098329094d7fee0dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93287b8840454786a4944835236be293" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38825 } } peers { permanent_uuid: "3f555fa910234b098329094d7fee0dfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37985 } } peers { permanent_uuid: "9b07c5e0fa79429c87a5563d331d6d24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40563 } }
I20260927 12:43:19.441246 13157 catalog_manager.cc:5495] T dbc9174d572f4ad896d35c434728dbfd P 3f555fa910234b098329094d7fee0dfb reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f555fa910234b098329094d7fee0dfb (127.0.98.1). New cstate: current_term: 1 leader_uuid: "3f555fa910234b098329094d7fee0dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93287b8840454786a4944835236be293" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f555fa910234b098329094d7fee0dfb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37985 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b07c5e0fa79429c87a5563d331d6d24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40563 } health_report { overall_health: UNKNOWN } } }
W20260927 12:43:19.454107 13357 tablet.cc:2369] T dbc9174d572f4ad896d35c434728dbfd P 3f555fa910234b098329094d7fee0dfb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 12:43:19.455257 13625 tablet.cc:2369] T dbc9174d572f4ad896d35c434728dbfd P 9b07c5e0fa79429c87a5563d331d6d24: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 12:43:19.465466 13491 tablet.cc:2369] T dbc9174d572f4ad896d35c434728dbfd P 93287b8840454786a4944835236be293: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:43:19.494464 13655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:19.489300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35422 (local address 127.0.98.1:37985)
0927 12:43:19.489481 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 12:43:19.489485 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:43:19.489503 (+ 18us) server_negotiation.cc:407] Connection header received
0927 12:43:19.489630 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:43:19.489634 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:43:19.489722 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:43:19.489847 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:43:19.490239 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:19.491039 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:19.492295 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:19.492625 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:19.492741 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:19.493160 (+ 419us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:43:19.493777 (+ 617us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:43:19.493900 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 12:43:19.494232 (+ 332us) server_negotiation.cc:299] Negotiation successful
0927 12:43:19.494297 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 12:43:19.514482 13578 raft_consensus.cc:1261] T dbc9174d572f4ad896d35c434728dbfd P 9b07c5e0fa79429c87a5563d331d6d24 [term 1 FOLLOWER]: Refusing update from remote peer 3f555fa910234b098329094d7fee0dfb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:43:19.515312 13667 consensus_queue.cc:1035] T dbc9174d572f4ad896d35c434728dbfd P 3f555fa910234b098329094d7fee0dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b07c5e0fa79429c87a5563d331d6d24" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:43:19.522852 13444 raft_consensus.cc:1261] T dbc9174d572f4ad896d35c434728dbfd P 93287b8840454786a4944835236be293 [term 1 FOLLOWER]: Refusing update from remote peer 3f555fa910234b098329094d7fee0dfb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:43:19.523491 13667 consensus_queue.cc:1035] T dbc9174d572f4ad896d35c434728dbfd P 3f555fa910234b098329094d7fee0dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "93287b8840454786a4944835236be293" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:43:19.550105 13157 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41730:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:43:19.550290 13157 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41730:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:43:19.552440 13673 mvcc.cc:205] Tried to move back new op lower bound from 7333941246000812032 to 7333941245700304896. Current Snapshot: MvccSnapshot[applied={T|T < 7333941246000812032 or (T in {7333941246000812032})}]
I20260927 12:43:19.552646 13676 mvcc.cc:205] Tried to move back new op lower bound from 7333941246000812032 to 7333941245700304896. Current Snapshot: MvccSnapshot[applied={T|T < 7333941246000812032}]
I20260927 12:43:19.553012 13157 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38a6c6e565764580ac353d604b8b057b: Sending DeleteTablet for 3 replicas of tablet dbc9174d572f4ad896d35c434728dbfd
I20260927 12:43:19.553238 13674 mvcc.cc:205] Tried to move back new op lower bound from 7333941246000812032 to 7333941245700304896. Current Snapshot: MvccSnapshot[applied={T|T < 7333941246000812032 or (T in {7333941246000812032})}]
I20260927 12:43:19.553774 13424 tablet_service.cc:1514] Processing DeleteTablet for tablet dbc9174d572f4ad896d35c434728dbfd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:43:19 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34696
I20260927 12:43:19.554052 13290 tablet_service.cc:1514] Processing DeleteTablet for tablet dbc9174d572f4ad896d35c434728dbfd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:43:19 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35410
I20260927 12:43:19.554823 13558 tablet_service.cc:1514] Processing DeleteTablet for tablet dbc9174d572f4ad896d35c434728dbfd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:43:19 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44926
I20260927 12:43:19.556211 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 13196
I20260927 12:43:19.565016 13684 tablet_replica.cc:330] T dbc9174d572f4ad896d35c434728dbfd P 9b07c5e0fa79429c87a5563d331d6d24: stopping tablet replica
I20260927 12:43:19.565407 13684 raft_consensus.cc:2229] T dbc9174d572f4ad896d35c434728dbfd P 9b07c5e0fa79429c87a5563d331d6d24 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:43:19.565788 13684 raft_consensus.cc:2258] T dbc9174d572f4ad896d35c434728dbfd P 9b07c5e0fa79429c87a5563d331d6d24 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:43:19.574985 13684 ts_tablet_manager.cc:1905] T dbc9174d572f4ad896d35c434728dbfd P 9b07c5e0fa79429c87a5563d331d6d24: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:43:19.577896 13684 ts_tablet_manager.cc:1918] T dbc9174d572f4ad896d35c434728dbfd P 9b07c5e0fa79429c87a5563d331d6d24: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:43:19.578001 13684 log.cc:1196] T dbc9174d572f4ad896d35c434728dbfd P 9b07c5e0fa79429c87a5563d331d6d24: Deleting WAL directory at /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790512840836255-392-0/minicluster-data/ts-2/wal/wals/dbc9174d572f4ad896d35c434728dbfd
I20260927 12:43:19.578307 13684 ts_tablet_manager.cc:1939] T dbc9174d572f4ad896d35c434728dbfd P 9b07c5e0fa79429c87a5563d331d6d24: Deleting consensus metadata
I20260927 12:43:19.578866 13682 tablet_replica.cc:330] T dbc9174d572f4ad896d35c434728dbfd P 93287b8840454786a4944835236be293: stopping tablet replica
I20260927 12:43:19.579632 13144 catalog_manager.cc:4841] TS 9b07c5e0fa79429c87a5563d331d6d24 (127.0.98.3:40563): tablet dbc9174d572f4ad896d35c434728dbfd (table test-table [id=8aa7dfc74d7f4942b99914748ccd16d9]) successfully deleted
I20260927 12:43:19.580438 13682 raft_consensus.cc:2229] T dbc9174d572f4ad896d35c434728dbfd P 93287b8840454786a4944835236be293 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:43:19.580824 13682 raft_consensus.cc:2258] T dbc9174d572f4ad896d35c434728dbfd P 93287b8840454786a4944835236be293 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260927 12:43:19.581314 13143 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37985 (error 108)
I20260927 12:43:19.581789 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 13358
I20260927 12:43:19.585547 13653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:19.585167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37985 (local address 127.0.0.1:35436)
0927 12:43:19.585373 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:19.585450 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
W20260927 12:43:19.585891 13143 catalog_manager.cc:4584] TS 3f555fa910234b098329094d7fee0dfb (127.0.98.1:37985): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dbc9174d572f4ad896d35c434728dbfd: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37985: connect: Connection refused (error 111)
I20260927 12:43:19.590476 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 13493
I20260927 12:43:19.591307 13653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:19.590822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38825 (local address 127.0.0.1:34708)
0927 12:43:19.591163 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:19.591217 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:38825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253}
I20260927 12:43:19.597672 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 13125
I20260927 12:43:19.606244 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 12:43:19.941113 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 12:43:19.941215 392 mini_ranger.cc:66] Stopping Ranger...
I20260927 12:43:20.286759 392 mini_ranger.cc:68] Stopped Ranger
2026-09-27 12:43:20.286 UTC [12192] LOG: received smart shutdown request
2026-09-27 12:43:20.290 UTC [12192] LOG: background worker "logical replication launcher" (PID 12201) exited with exit code 1
2026-09-27 12:43:20.290 UTC [12196] LOG: shutting down
2026-09-27 12:43:20.465 UTC [12192] LOG: database system is shut down
2026-09-27T12:43:20Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48472 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512840836255-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:43:20 dist-test-slave-bnc1 krb5kdc[13701](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:43:20 dist-test-slave-bnc1 krb5kdc[13701](info): set up 2 sockets
Sep 27 12:43:20 dist-test-slave-bnc1 krb5kdc[13701](info): commencing operation
krb5kdc: starting...
W20260927 12:43:21.645823 392 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512840836255-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512840836255-392-0/krb5kdc/test-user.keytab.
Sep 27 12:43:21 dist-test-slave-bnc1 krb5kdc[13701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:43:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:43:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:43:21.769224 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512840836255-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512840836255-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512840836255-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45903
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45903,127.0.98.61:42127,127.0.98.60:44311
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:43:21.875619 13714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:43:21.875941 13714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:43:21.876021 13714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:43:21.879992 13714 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:43:21.880067 13714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:43:21.880105 13714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:43:21.880162 13714 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:43:21.880198 13714 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:43:21.885167 13714 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512840836255-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512840836255-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45903,127.0.98.61:42127,127.0.98.60:44311
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512840836255-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:43:21.886232 13714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:43:21.887190 13714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:43:21.895336 13719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:43:21.895354 13720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:43:21.895371 13722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:43:21.895354 13714 server_base.cc:1034] running on GCE node
I20260927 12:43:21.896163 13714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:43:21.897169 13714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:43:21.898370 13714 hybrid_clock.cc:648] HybridClock initialized: now 1790513001898360 us; error 30 us; skew 500 ppm
Sep 27 12:43:21 dist-test-slave-bnc1 krb5kdc[13701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513001, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:43:21.901043 13714 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:43:21.901960 13714 webserver.cc:460] Webserver started at http://127.0.98.62:41577/ using document root <none> and password file <none>
I20260927 12:43:21.902491 13714 fs_manager.cc:359] Metadata directory not provided
I20260927 12:43:21.902578 13714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:43:21.902794 13714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:43:21.904745 13714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512840836255-392-0/minicluster-data/master-0/data/instance:
uuid: "4d3f3a636fe040cabc19b7695316a42b"
format_stamp: "Formatted at 2026-09-27 12:43:21 on dist-test-slave-bnc1"
server_key: "785b954504f98d6901d669189908aa4d"
server_key_iv: "cb438f51a292b4bd4c96cd65953fafce"
server_key_version: "encryptionkey@0"
I20260927 12:43:21.905279 13714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512840836255-392-0/minicluster-data/master-0/wal/instance:
uuid: "4d3f3a636fe040cabc19b7695316a42b"
format_stamp: "Formatted at 2026-09-27 12:43:21 on dist-test-slave-bnc1"
server_key: "785b954504f98d6901d669189908aa4d"
server_key_iv: "cb438f51a292b4bd4c96cd65953fafce"
server_key_version: "encryptionkey@0"
I20260927 12:43:21.908684 13714 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 12:43:21.910995 13729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:43:21.912200 13714 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 12:43:21.912336 13714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512840836255-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512840836255-392-0/minicluster-data/master-0/wal
uuid: "4d3f3a636fe040cabc19b7695316a42b"
format_stamp: "Formatted at 2026-09-27 12:43:21 on dist-test-slave-bnc1"
server_key: "785b954504f98d6901d669189908aa4d"
server_key_iv: "cb438f51a292b4bd4c96cd65953fafce"
server_key_version: "encryptionkey@0"
I20260927 12:43:21.912438 13714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512840836255-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512840836255-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512840836255-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:43:21.931170 13714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:43:21.934423 13714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:43:21.934621 13714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:43:21.941947 13714 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45903
I20260927 12:43:21.941973 13781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45903 every 8 connection(s)
I20260927 12:43:21.942883 13714 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:43:21.944620 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 13714
I20260927 12:43:21.944721 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512840836255-392-0/minicluster-data/master-0/wal/instance
I20260927 12:43:21.945036 392 external_mini_cluster.cc:1428] Setting key 5271bf6f2ed3a7432bfc4332b3228067
I20260927 12:43:21.945819 13782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:43:21.949465 13782 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45903 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 27 12:43:21 dist-test-slave-bnc1 krb5kdc[13701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513001, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:43:21.965435 13785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:21.951884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45903 (local address 127.0.0.1:58946)
0927 12:43:21.952503 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:21.952565 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0927 12:43:21.953436 (+ 871us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:43:21.954215 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:43:21.954220 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:43:21.954647 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:43:21.954936 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:21.954949 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:21.956284 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:21.956290 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:21.957261 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:21.957271 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:21.958085 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:21.958088 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:21.958166 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:21.958750 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:43:21.958765 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:43:21.960616 (+ 1851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:43:21.962664 (+ 2048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:43:21.962676 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:43:21.962687 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:43:21.963001 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:43:21.963320 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:43:21.963324 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:43:21.963326 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:43:21.963460 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:43:21.964004 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:43:21.964021 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:43:21.964136 (+ 115us) client_negotiation.cc:769] Sending connection context
0927 12:43:21.964302 (+ 166us) client_negotiation.cc:240] Negotiation successful
0927 12:43:21.964707 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"mutex_wait_us":105,"thread_start_us":175,"threads_started":1}
I20260927 12:43:21.965437 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:21.952916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58946 (local address 127.0.98.62:45903)
0927 12:43:21.953190 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0927 12:43:21.953198 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 12:43:21.953230 (+ 32us) server_negotiation.cc:407] Connection header received
0927 12:43:21.953801 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:43:21.953807 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:43:21.954030 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:43:21.954218 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:43:21.955095 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:21.956149 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:21.957389 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:21.957946 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:21.958067 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:21.960762 (+ 2695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:43:21.960767 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:43:21.960777 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:43:21.960797 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:43:21.962546 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:21.963114 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:21.963118 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:21.963119 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:21.963215 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:21.963558 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:21.963560 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:21.963561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:21.963896 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:43:21.963994 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 12:43:21.964358 (+ 364us) server_negotiation.cc:299] Negotiation successful
0927 12:43:21.964635 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":172,"thread_start_us":97,"threads_started":1}
I20260927 12:43:21.968060 13782 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42127 } has no permanent_uuid. Determining permanent_uuid...
I20260927 12:43:21.968820 13785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:21.968565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42127 (local address 127.0.0.1:59304)
0927 12:43:21.968688 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:21.968769 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 12:43:21.969076 13730 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42127: connect: Connection refused (error 111)
I20260927 12:43:21.969907 13785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:21.969694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42127 (local address 127.0.0.1:59308)
0927 12:43:21.969817 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:21.969860 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260927 12:43:21.970060 13782 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42127: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42127: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512840836255-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512840836255-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260927 12:43:21.972481 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512840836255-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512840836255-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512840836255-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512840836255-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:42127
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512840836255-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45903,127.0.98.61:42127,127.0.98.60:44311
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 12:43:22.019297 13782 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42127 } attempt: 1
I20260927 12:43:22.020218 13785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:22.019881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42127 (local address 127.0.0.1:59310)
0927 12:43:22.020053 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:22.020126 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260927 12:43:22.021000 13785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:22.020771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42127 (local address 127.0.0.1:59316)
0927 12:43:22.020880 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:22.020934 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 12:43:22.021158 13782 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42127: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42127: connect: Connection refused (error 111)
W20260927 12:43:22.082171 13789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:43:22.082430 13789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:43:22.082469 13789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:43:22.086493 13789 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:43:22.086547 13789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:43:22.086565 13789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:43:22.086582 13789 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:43:22.086596 13789 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:43:22.089367 13782 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42127 } attempt: 2
I20260927 12:43:22.090305 13785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:22.089951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42127 (local address 127.0.0.1:59322)
0927 12:43:22.090135 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:22.090208 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260927 12:43:22.091022 13785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:22.090807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42127 (local address 127.0.0.1:59326)
0927 12:43:22.090912 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:22.090962 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 12:43:22.091193 13782 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42127: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42127: connect: Connection refused (error 111)
I20260927 12:43:22.092409 13789 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512840836255-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512840836255-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45903,127.0.98.61:42127,127.0.98.60:44311
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512840836255-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:42127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512840836255-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13789
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512840836255-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:43:22.093447 13789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:43:22.094477 13789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:43:22.102805 13795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:43:22.102792 13794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:43:22.102861 13789 server_base.cc:1034] running on GCE node
W20260927 12:43:22.102795 13797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:43:22.103477 13789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:43:22.104460 13789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:43:22.105644 13789 hybrid_clock.cc:648] HybridClock initialized: now 1790513002105621 us; error 26 us; skew 500 ppm
Sep 27 12:43:22 dist-test-slave-bnc1 krb5kdc[13701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513002, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:43:22.108248 13789 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260927 12:43:22.109033 13789 webserver.cc:460] Webserver started at http://127.0.98.61:40335/ using document root <none> and password file <none>
I20260927 12:43:22.109560 13789 fs_manager.cc:359] Metadata directory not provided
I20260927 12:43:22.109706 13789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:43:22.109910 13789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:43:22.111806 13789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512840836255-392-0/minicluster-data/master-1/data/instance:
uuid: "04c92d78793443b28b58aac89e6e65c1"
format_stamp: "Formatted at 2026-09-27 12:43:22 on dist-test-slave-bnc1"
server_key: "447de9771a0e0d3946d197f54a771ff6"
server_key_iv: "5b8f793abc382257023c561611780a03"
server_key_version: "encryptionkey@0"
I20260927 12:43:22.112336 13789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512840836255-392-0/minicluster-data/master-1/wal/instance:
uuid: "04c92d78793443b28b58aac89e6e65c1"
format_stamp: "Formatted at 2026-09-27 12:43:22 on dist-test-slave-bnc1"
server_key: "447de9771a0e0d3946d197f54a771ff6"
server_key_iv: "5b8f793abc382257023c561611780a03"
server_key_version: "encryptionkey@0"
I20260927 12:43:22.115985 13789 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 12:43:22.118464 13804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:43:22.119632 13789 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260927 12:43:22.119773 13789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512840836255-392-0/minicluster-data/master-1/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512840836255-392-0/minicluster-data/master-1/wal
uuid: "04c92d78793443b28b58aac89e6e65c1"
format_stamp: "Formatted at 2026-09-27 12:43:22 on dist-test-slave-bnc1"
server_key: "447de9771a0e0d3946d197f54a771ff6"
server_key_iv: "5b8f793abc382257023c561611780a03"
server_key_version: "encryptionkey@0"
I20260927 12:43:22.119880 13789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512840836255-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512840836255-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512840836255-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:43:22.137209 13789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:43:22.139892 13789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:43:22.140094 13789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:43:22.146932 13789 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:42127
I20260927 12:43:22.146962 13856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:42127 every 8 connection(s)
I20260927 12:43:22.147786 13789 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512840836255-392-0/minicluster-data/master-1/data/info.pb
I20260927 12:43:22.148063 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 13789
I20260927 12:43:22.148197 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512840836255-392-0/minicluster-data/master-1/wal/instance
I20260927 12:43:22.148523 392 external_mini_cluster.cc:1428] Setting key 6e57c35d302427136cfbbddf605d35dc
I20260927 12:43:22.150547 13857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:43:22.154235 13857 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45903 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 27 12:43:22 dist-test-slave-bnc1 krb5kdc[13701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513002, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:43:22.168265 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:22.156048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58950 (local address 127.0.98.62:45903)
0927 12:43:22.156182 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 12:43:22.156185 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:43:22.157025 (+ 840us) server_negotiation.cc:407] Connection header received
0927 12:43:22.158188 (+ 1163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:43:22.158192 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:43:22.158251 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:43:22.158373 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:43:22.159451 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:22.159980 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:22.161112 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:22.161361 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:22.161475 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:22.164370 (+ 2895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:43:22.164373 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:43:22.164375 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:43:22.164398 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:43:22.166427 (+ 2029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:22.166863 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:22.166865 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:22.166866 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:22.166936 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:22.167215 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:22.167227 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:22.167228 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:22.167406 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:43:22.167481 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 12:43:22.168039 (+ 558us) server_negotiation.cc:299] Negotiation successful
0927 12:43:22.168147 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 12:43:22.169044 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:22.156429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45903 (local address 127.0.0.1:58950)
0927 12:43:22.156883 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:22.156924 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 12:43:22.157968 (+ 1044us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:43:22.158535 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:43:22.158542 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:43:22.158861 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:43:22.159306 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:22.159318 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:22.160105 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:22.160108 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:22.160985 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:22.160994 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:22.161519 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:22.161523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:22.161708 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:22.162349 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:43:22.162372 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:43:22.164185 (+ 1813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:43:22.166526 (+ 2341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:43:22.166531 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:43:22.166542 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:43:22.166787 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:43:22.167014 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:43:22.167016 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:43:22.167017 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:43:22.167141 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:43:22.167517 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:43:22.167526 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:43:22.167767 (+ 241us) client_negotiation.cc:769] Sending connection context
0927 12:43:22.167952 (+ 185us) client_negotiation.cc:240] Negotiation successful
0927 12:43:22.168213 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":106,"threads_started":1}
I20260927 12:43:22.170419 13857 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42127 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512840836255-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512840836255-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260927 12:43:22.172660 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512840836255-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512840836255-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512840836255-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512840836255-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:44311
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512840836255-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45903,127.0.98.61:42127,127.0.98.60:44311
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 12:43:22 dist-test-slave-bnc1 krb5kdc[13701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513002, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 12:43:22.179394 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:22.170904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42127 (local address 127.0.0.1:59340)
0927 12:43:22.171012 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:22.171027 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:43:22.171094 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:43:22.171693 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:43:22.171696 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:43:22.171878 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:43:22.172006 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:22.172017 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:22.173094 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:22.173100 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:22.173711 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:22.173719 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:22.174293 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:22.174296 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:22.174382 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:22.174846 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:43:22.174860 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:43:22.176014 (+ 1154us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:43:22.177948 (+ 1934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:43:22.177951 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:43:22.177953 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:43:22.178133 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:43:22.178414 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:43:22.178418 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:43:22.178420 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:43:22.178498 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:43:22.178914 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:43:22.178916 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:43:22.178992 (+ 76us) client_negotiation.cc:769] Sending connection context
0927 12:43:22.179107 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 12:43:22.179225 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 12:43:22.179456 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:22.171018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59340 (local address 127.0.98.61:42127)
0927 12:43:22.171375 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0927 12:43:22.171384 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 12:43:22.171401 (+ 17us) server_negotiation.cc:407] Connection header received
0927 12:43:22.171432 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:43:22.171436 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:43:22.171570 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:43:22.171712 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:43:22.172122 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:22.172986 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:22.173839 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:22.174182 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:22.174297 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:22.176157 (+ 1860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:43:22.176173 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:43:22.176182 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:43:22.176201 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:43:22.177829 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:22.178237 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:22.178243 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:22.178245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:22.178324 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:22.178582 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:22.178587 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:22.178589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:22.178814 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:43:22.178948 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 12:43:22.179184 (+ 236us) server_negotiation.cc:299] Negotiation successful
0927 12:43:22.179313 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":172,"threads_started":1}
I20260927 12:43:22.181139 13857 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44311 } has no permanent_uuid. Determining permanent_uuid...
I20260927 12:43:22.182015 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:22.181698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44311 (local address 127.0.0.1:46146)
0927 12:43:22.181838 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:22.181926 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 12:43:22.182220 13808 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44311: connect: Connection refused (error 111)
I20260927 12:43:22.182376 13782 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42127 } attempt: 3
I20260927 12:43:22.183465 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:22.183246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44311 (local address 127.0.0.1:46154)
0927 12:43:22.183354 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:22.183405 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 12:43:22.183888 13857 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44311: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44311: connect: Connection refused (error 111)
Sep 27 12:43:22 dist-test-slave-bnc1 krb5kdc[13701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513001, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 12:43:22.191047 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:22.182878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59342 (local address 127.0.98.61:42127)
0927 12:43:22.182979 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0927 12:43:22.182983 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:43:22.182995 (+ 12us) server_negotiation.cc:407] Connection header received
0927 12:43:22.183337 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:43:22.183340 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:43:22.183373 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:43:22.183458 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:43:22.183867 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:22.184424 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:22.185169 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:22.186003 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:22.186114 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:22.188051 (+ 1937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:43:22.188054 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:43:22.188057 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:43:22.188078 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:43:22.189473 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:22.189939 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:22.189943 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:22.189947 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:22.190018 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:22.190291 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:22.190296 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:22.190297 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:22.190456 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:43:22.190560 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 12:43:22.190819 (+ 259us) server_negotiation.cc:299] Negotiation successful
0927 12:43:22.190920 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 12:43:22.191048 13785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:22.182788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42127 (local address 127.0.0.1:59342)
0927 12:43:22.182906 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:22.182921 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:43:22.183007 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:43:22.183478 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:43:22.183481 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:43:22.183665 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:43:22.183757 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:22.183763 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:22.184539 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:22.184543 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:22.185057 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:22.185064 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:22.186115 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:22.186119 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:22.186225 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:22.186727 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:43:22.186738 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:43:22.187923 (+ 1185us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:43:22.189616 (+ 1693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:43:22.189622 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:43:22.189627 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:43:22.189834 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:43:22.190111 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:43:22.190115 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:43:22.190117 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:43:22.190200 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:43:22.190552 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:43:22.190555 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:43:22.190655 (+ 100us) client_negotiation.cc:769] Sending connection context
0927 12:43:22.190762 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 12:43:22.190895 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260927 12:43:22.191735 13782 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44311 } has no permanent_uuid. Determining permanent_uuid...
I20260927 12:43:22.192395 13785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:22.192175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44311 (local address 127.0.0.1:46160)
0927 12:43:22.192274 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:22.192342 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 12:43:22.193055 13785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:22.192824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44311 (local address 127.0.0.1:46162)
0927 12:43:22.192964 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:22.193006 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260927 12:43:22.193193 13782 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44311: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44311: connect: Connection refused (error 111)
I20260927 12:43:22.224342 13782 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44311 } attempt: 1
I20260927 12:43:22.225139 13785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:22.224870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44311 (local address 127.0.0.1:46172)
0927 12:43:22.225009 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:22.225067 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 12:43:22.225901 13785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:22.225691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44311 (local address 127.0.0.1:46188)
0927 12:43:22.225794 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:22.225848 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260927 12:43:22.226027 13782 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44311: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44311: connect: Connection refused (error 111)
I20260927 12:43:22.233141 13857 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44311 } attempt: 1
I20260927 12:43:22.233902 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:22.233623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44311 (local address 127.0.0.1:46196)
0927 12:43:22.233763 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:22.233824 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 12:43:22.234580 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:22.234369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44311 (local address 127.0.0.1:46206)
0927 12:43:22.234471 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:22.234522 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 12:43:22.234712 13857 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44311: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44311: connect: Connection refused (error 111)
W20260927 12:43:22.282855 13863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:43:22.283103 13863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:43:22.283143 13863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:43:22.287068 13863 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:43:22.287123 13863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:43:22.287142 13863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:43:22.287159 13863 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:43:22.287174 13863 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:43:22.292201 13863 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512840836255-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512840836255-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45903,127.0.98.61:42127,127.0.98.60:44311
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512840836255-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:44311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512840836255-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13863
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512840836255-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:43:22.293134 13863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:43:22.294108 13863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 12:43:22.301188 13782 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44311 } attempt: 2
I20260927 12:43:22.301987 13785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:22.301695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44311 (local address 127.0.0.1:46210)
0927 12:43:22.301857 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:22.301914 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 12:43:22.302237 13872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:43:22.302377 13863 server_base.cc:1034] running on GCE node
W20260927 12:43:22.302239 13869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:43:22.302649 13870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:43:22.302866 13857 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44311 } attempt: 2
I20260927 12:43:22.302942 13785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:22.302674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44311 (local address 127.0.0.1:46224)
0927 12:43:22.302813 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:22.302870 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 12:43:22.303066 13782 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44311: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44311: connect: Connection refused (error 111)
I20260927 12:43:22.303551 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:22.303264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44311 (local address 127.0.0.1:46240)
0927 12:43:22.303407 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:22.303463 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 12:43:22.303601 13863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:43:22.304226 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:22.303990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44311 (local address 127.0.0.1:46256)
0927 12:43:22.304117 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:22.304166 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 12:43:22.304404 13857 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44311: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44311: connect: Connection refused (error 111)
I20260927 12:43:22.304822 13863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:43:22.306005 13863 hybrid_clock.cc:648] HybridClock initialized: now 1790513002305989 us; error 25 us; skew 500 ppm
Sep 27 12:43:22 dist-test-slave-bnc1 krb5kdc[13701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513002, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:43:22.308557 13863 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260927 12:43:22.309409 13863 webserver.cc:460] Webserver started at http://127.0.98.60:33493/ using document root <none> and password file <none>
I20260927 12:43:22.309983 13863 fs_manager.cc:359] Metadata directory not provided
I20260927 12:43:22.310070 13863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:43:22.310278 13863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:43:22.312201 13863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512840836255-392-0/minicluster-data/master-2/data/instance:
uuid: "37f12cb097cd4887b3ba4da5921f5197"
format_stamp: "Formatted at 2026-09-27 12:43:22 on dist-test-slave-bnc1"
server_key: "dee58ee1e50966a2f25aa2d80ebe4951"
server_key_iv: "6b807e15cb8b120d2a14fe991ba1398d"
server_key_version: "encryptionkey@0"
I20260927 12:43:22.312779 13863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512840836255-392-0/minicluster-data/master-2/wal/instance:
uuid: "37f12cb097cd4887b3ba4da5921f5197"
format_stamp: "Formatted at 2026-09-27 12:43:22 on dist-test-slave-bnc1"
server_key: "dee58ee1e50966a2f25aa2d80ebe4951"
server_key_iv: "6b807e15cb8b120d2a14fe991ba1398d"
server_key_version: "encryptionkey@0"
I20260927 12:43:22.316236 13863 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 12:43:22.318511 13879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:43:22.319718 13863 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 12:43:22.319860 13863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512840836255-392-0/minicluster-data/master-2/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512840836255-392-0/minicluster-data/master-2/wal
uuid: "37f12cb097cd4887b3ba4da5921f5197"
format_stamp: "Formatted at 2026-09-27 12:43:22 on dist-test-slave-bnc1"
server_key: "dee58ee1e50966a2f25aa2d80ebe4951"
server_key_iv: "6b807e15cb8b120d2a14fe991ba1398d"
server_key_version: "encryptionkey@0"
I20260927 12:43:22.319958 13863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512840836255-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512840836255-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512840836255-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:43:22.343503 13863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:43:22.346451 13863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:43:22.346688 13863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:43:22.353809 13863 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:44311
I20260927 12:43:22.353863 13931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:44311 every 8 connection(s)
I20260927 12:43:22.354741 13863 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512840836255-392-0/minicluster-data/master-2/data/info.pb
I20260927 12:43:22.357388 13932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:43:22.358451 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 13863
I20260927 12:43:22.358580 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790512840836255-392-0/minicluster-data/master-2/wal/instance
I20260927 12:43:22.358911 392 external_mini_cluster.cc:1428] Setting key f4cfa4cbcf234c88d87088f22494637b
I20260927 12:43:22.361718 13932 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45903 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 12:43:22 dist-test-slave-bnc1 krb5kdc[13701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 12:43:22 dist-test-slave-bnc1 krb5kdc[13701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513002, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:43:22.373025 13935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:22.360231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46260 (local address 127.0.98.60:44311)
0927 12:43:22.360662 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0927 12:43:22.360676 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 12:43:22.360708 (+ 32us) server_negotiation.cc:407] Connection header received
0927 12:43:22.361302 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:43:22.361317 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:43:22.361696 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:43:22.362020 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:43:22.362847 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:22.364082 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:22.364878 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:22.365307 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:22.365483 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:22.368063 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:43:22.368072 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:43:22.368085 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:43:22.368132 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:43:22.370028 (+ 1896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:22.370529 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:22.370538 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:22.370545 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:22.370680 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:22.371020 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:22.371023 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:22.371025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:22.371525 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:43:22.371749 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0927 12:43:22.372019 (+ 270us) server_negotiation.cc:299] Negotiation successful
0927 12:43:22.372292 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"spinlock_wait_cycles":1408,"thread_start_us":129,"threads_started":1}
I20260927 12:43:22.373581 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:22.363383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58956 (local address 127.0.98.62:45903)
0927 12:43:22.363544 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 12:43:22.363548 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:43:22.363895 (+ 347us) server_negotiation.cc:407] Connection header received
0927 12:43:22.364031 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:43:22.364034 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:43:22.364084 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:43:22.364182 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:43:22.365497 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:22.366147 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:22.367028 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:22.367261 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:22.367377 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:22.369993 (+ 2616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:43:22.369997 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:43:22.370000 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:43:22.370026 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:43:22.371728 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:22.372242 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:22.372246 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:22.372247 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:22.372330 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:22.372607 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:22.372612 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:22.372613 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:22.372801 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:43:22.372918 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 12:43:22.373294 (+ 376us) server_negotiation.cc:299] Negotiation successful
0927 12:43:22.373413 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 12:43:22.373605 13936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:22.363314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45903 (local address 127.0.0.1:58956)
0927 12:43:22.363765 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:22.363809 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 12:43:22.363923 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:43:22.364939 (+ 1016us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:43:22.364944 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:43:22.365247 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:43:22.365373 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:22.365378 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:22.366268 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:22.366272 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:22.366888 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:22.366896 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:22.367409 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:22.367413 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:22.367546 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:22.368110 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:43:22.368125 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:43:22.369844 (+ 1719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:43:22.371836 (+ 1992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:43:22.371847 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:43:22.371855 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:43:22.372150 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:43:22.372424 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:43:22.372430 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:43:22.372433 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:43:22.372513 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:43:22.372916 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:43:22.372923 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:43:22.373035 (+ 112us) client_negotiation.cc:769] Sending connection context
0927 12:43:22.373232 (+ 197us) client_negotiation.cc:240] Negotiation successful
0927 12:43:22.373414 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":160,"threads_started":1}
I20260927 12:43:22.374938 13932 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42127 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 12:43:22 dist-test-slave-bnc1 krb5kdc[13701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513002, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 27 12:43:22 dist-test-slave-bnc1 krb5kdc[13701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:43:22.383790 13936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:22.375413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42127 (local address 127.0.0.1:59350)
0927 12:43:22.375654 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:22.375665 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 12:43:22.375733 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:43:22.375952 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:43:22.375953 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:43:22.376074 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:43:22.376151 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:22.376156 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:22.377023 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:22.377026 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:22.377692 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:22.377701 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:22.378190 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:22.378194 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:22.378300 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:22.378751 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:43:22.378768 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:43:22.380019 (+ 1251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:43:22.382093 (+ 2074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:43:22.382096 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:43:22.382100 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:43:22.382335 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:43:22.382593 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:43:22.382596 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:43:22.382597 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:43:22.382687 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:43:22.383238 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:43:22.383244 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:43:22.383345 (+ 101us) client_negotiation.cc:769] Sending connection context
0927 12:43:22.383513 (+ 168us) client_negotiation.cc:240] Negotiation successful
0927 12:43:22.383636 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260927 12:43:22.383809 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:22.375508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59350 (local address 127.0.98.61:42127)
0927 12:43:22.375615 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 12:43:22.375618 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:43:22.375734 (+ 116us) server_negotiation.cc:407] Connection header received
0927 12:43:22.375811 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:43:22.375814 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:43:22.375860 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:43:22.375936 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:43:22.376265 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:22.376902 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:22.377798 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:22.378088 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:22.378165 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:22.380178 (+ 2013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:43:22.380181 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:43:22.380184 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:43:22.380210 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:43:22.381983 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:22.382435 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:22.382438 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:22.382439 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:22.382507 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:22.382826 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:22.382831 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:22.382834 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:22.383144 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:43:22.383243 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 12:43:22.383573 (+ 330us) server_negotiation.cc:299] Negotiation successful
0927 12:43:22.383677 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 12:43:22.384780 13932 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44311 } has no permanent_uuid. Determining permanent_uuid...
I20260927 12:43:22.386034 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:22.375357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58970 (local address 127.0.98.62:45903)
0927 12:43:22.375456 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0927 12:43:22.375460 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:43:22.375535 (+ 75us) server_negotiation.cc:407] Connection header received
0927 12:43:22.375645 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:43:22.375648 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:43:22.375715 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:43:22.375797 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:43:22.376567 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:22.377099 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:22.377960 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:22.378195 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:22.378293 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:22.381118 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:43:22.381122 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:43:22.381125 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:43:22.381152 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:43:22.383843 (+ 2691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:22.384294 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:22.384298 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:22.384300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:22.384381 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:22.384649 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:22.384654 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:22.384655 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:22.384847 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:43:22.385091 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0927 12:43:22.385724 (+ 633us) server_negotiation.cc:299] Negotiation successful
0927 12:43:22.385869 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
Sep 27 12:43:22 dist-test-slave-bnc1 krb5kdc[13701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513002, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 12:43:22 dist-test-slave-bnc1 krb5kdc[13701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 12:43:22.393157 13936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:22.385249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44311 (local address 127.0.0.1:46274)
0927 12:43:22.385450 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:22.385465 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:43:22.385545 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:43:22.385830 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:43:22.385836 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:43:22.385984 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:43:22.386072 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:22.386079 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:22.387155 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:22.387158 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:22.387632 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:22.387638 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:22.388213 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:22.388216 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:22.388272 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:22.388687 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:43:22.388697 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:43:22.390020 (+ 1323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:43:22.391756 (+ 1736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:43:22.391762 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:43:22.391764 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:43:22.391954 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:43:22.392240 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:43:22.392242 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:43:22.392244 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:43:22.392325 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:43:22.392666 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:43:22.392669 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:43:22.392744 (+ 75us) client_negotiation.cc:769] Sending connection context
0927 12:43:22.392864 (+ 120us) client_negotiation.cc:240] Negotiation successful
0927 12:43:22.392990 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136}
I20260927 12:43:22.393158 13935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:22.385308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46274 (local address 127.0.98.60:44311)
0927 12:43:22.385409 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0927 12:43:22.385413 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:43:22.385505 (+ 92us) server_negotiation.cc:407] Connection header received
0927 12:43:22.385676 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:43:22.385679 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:43:22.385723 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:43:22.385792 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:43:22.386191 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:22.387041 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:22.387715 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:22.388102 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:22.388223 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:22.390119 (+ 1896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:43:22.390122 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:43:22.390125 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:43:22.390148 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:43:22.391655 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:22.392046 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:22.392052 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:22.392054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:22.392137 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:22.392415 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:22.392418 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:22.392420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:22.392574 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:43:22.392675 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 12:43:22.392906 (+ 231us) server_negotiation.cc:299] Negotiation successful
0927 12:43:22.393045 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":43}
I20260927 12:43:22.395561 13857 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44311 } attempt: 3
I20260927 12:43:22.396546 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:22.387152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59352 (local address 127.0.98.61:42127)
0927 12:43:22.387247 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0927 12:43:22.387251 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:43:22.387325 (+ 74us) server_negotiation.cc:407] Connection header received
0927 12:43:22.387435 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:43:22.387438 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:43:22.387475 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:43:22.387566 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:43:22.388245 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:22.388919 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:22.390008 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:22.390340 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:22.390438 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:22.392533 (+ 2095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:43:22.392537 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:43:22.392539 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:43:22.392562 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:43:22.394592 (+ 2030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:22.394978 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:22.394981 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:22.394983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:22.395065 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:22.395306 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:22.395309 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:22.395311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:22.395516 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:43:22.395739 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0927 12:43:22.396297 (+ 558us) server_negotiation.cc:299] Negotiation successful
0927 12:43:22.396410 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260927 12:43:22.396930 13932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37f12cb097cd4887b3ba4da5921f5197: Bootstrap starting.
I20260927 12:43:22.399901 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 12:43:22.400177 13932 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37f12cb097cd4887b3ba4da5921f5197: Neither blocks nor log segments found. Creating new log.
Sep 27 12:43:22 dist-test-slave-bnc1 krb5kdc[13701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513002, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 12:43:22.401126 13932 log.cc:824] T 00000000000000000000000000000000 P 37f12cb097cd4887b3ba4da5921f5197: Log is configured to *not* fsync() on all Append() calls
I20260927 12:43:22.403378 13932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37f12cb097cd4887b3ba4da5921f5197: No bootstrap required, opened a new log
I20260927 12:43:22.404321 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:22.395946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44311 (local address 127.0.0.1:46276)
0927 12:43:22.396106 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:22.396117 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 12:43:22.396197 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:43:22.396523 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:43:22.396526 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:43:22.396718 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:43:22.396803 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:22.396818 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:22.397703 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:22.397708 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:22.398328 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:22.398337 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:22.398856 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:22.398860 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:22.398942 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:22.399456 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:43:22.399469 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:43:22.400809 (+ 1340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:43:22.402713 (+ 1904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:43:22.402716 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:43:22.402718 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:43:22.402968 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:43:22.403291 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:43:22.403294 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:43:22.403296 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:43:22.403396 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:43:22.403865 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:43:22.403868 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:43:22.403955 (+ 87us) client_negotiation.cc:769] Sending connection context
0927 12:43:22.404032 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 12:43:22.404182 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97}
I20260927 12:43:22.404809 13935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:22.395994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46276 (local address 127.0.98.60:44311)
0927 12:43:22.396354 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0927 12:43:22.396357 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:43:22.396366 (+ 9us) server_negotiation.cc:407] Connection header received
0927 12:43:22.396392 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:43:22.396393 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:43:22.396423 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:43:22.396507 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:43:22.396924 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:22.397618 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:22.398442 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:22.398734 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:22.398839 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:22.400945 (+ 2106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:43:22.400951 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:43:22.400955 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:43:22.400979 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:43:22.402609 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:22.403102 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:22.403106 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:22.403107 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:22.403188 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:22.403517 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:22.403520 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:22.403521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:22.403749 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:43:22.404494 (+ 745us) server_negotiation.cc:1035] Waiting for connection context
0927 12:43:22.404562 (+ 68us) server_negotiation.cc:299] Negotiation successful
0927 12:43:22.404660 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278}
I20260927 12:43:22.405613 13782 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44311 } attempt: 3
I20260927 12:43:22.407636 13857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04c92d78793443b28b58aac89e6e65c1: Bootstrap starting.
I20260927 12:43:22.407781 13932 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37f12cb097cd4887b3ba4da5921f5197 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f3a636fe040cabc19b7695316a42b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45903 } } peers { permanent_uuid: "04c92d78793443b28b58aac89e6e65c1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42127 } } peers { permanent_uuid: "37f12cb097cd4887b3ba4da5921f5197" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44311 } }
I20260927 12:43:22.408011 13932 raft_consensus.cc:374] T 00000000000000000000000000000000 P 37f12cb097cd4887b3ba4da5921f5197 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:43:22.408066 13932 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37f12cb097cd4887b3ba4da5921f5197 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37f12cb097cd4887b3ba4da5921f5197, State: Initialized, Role: FOLLOWER
I20260927 12:43:22.408537 13932 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37f12cb097cd4887b3ba4da5921f5197 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f3a636fe040cabc19b7695316a42b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45903 } } peers { permanent_uuid: "04c92d78793443b28b58aac89e6e65c1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42127 } } peers { permanent_uuid: "37f12cb097cd4887b3ba4da5921f5197" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44311 } }
I20260927 12:43:22.409013 13946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37f12cb097cd4887b3ba4da5921f5197 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f3a636fe040cabc19b7695316a42b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45903 } } peers { permanent_uuid: "04c92d78793443b28b58aac89e6e65c1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42127 } } peers { permanent_uuid: "37f12cb097cd4887b3ba4da5921f5197" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44311 } } }
I20260927 12:43:22.409210 13946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37f12cb097cd4887b3ba4da5921f5197 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 12:43:22.409937 13932 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37f12cb097cd4887b3ba4da5921f5197 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:43:22.410331 13857 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04c92d78793443b28b58aac89e6e65c1: Neither blocks nor log segments found. Creating new log.
I20260927 12:43:22.411233 13857 log.cc:824] T 00000000000000000000000000000000 P 04c92d78793443b28b58aac89e6e65c1: Log is configured to *not* fsync() on all Append() calls
I20260927 12:43:22.413798 13946 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37f12cb097cd4887b3ba4da5921f5197 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:43:22.413938 13946 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37f12cb097cd4887b3ba4da5921f5197 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f3a636fe040cabc19b7695316a42b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45903 } } peers { permanent_uuid: "04c92d78793443b28b58aac89e6e65c1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42127 } } peers { permanent_uuid: "37f12cb097cd4887b3ba4da5921f5197" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44311 } }
I20260927 12:43:22.414781 13946 leader_election.cc:288] T 00000000000000000000000000000000 P 37f12cb097cd4887b3ba4da5921f5197 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3f3a636fe040cabc19b7695316a42b (127.0.98.62:45903), 04c92d78793443b28b58aac89e6e65c1 (127.0.98.61:42127)
I20260927 12:43:22.416070 13945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:22.403236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46288 (local address 127.0.98.60:44311)
0927 12:43:22.403503 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0927 12:43:22.403511 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 12:43:22.406119 (+ 2608us) server_negotiation.cc:407] Connection header received
0927 12:43:22.406259 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:43:22.406262 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:43:22.406301 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:43:22.406370 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:43:22.407252 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:22.408086 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:22.409154 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:22.409491 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:22.409636 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:22.411504 (+ 1868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:43:22.411509 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:43:22.411512 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:43:22.411538 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:43:22.413376 (+ 1838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:22.413891 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:22.413894 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:22.413896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:22.413976 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:22.414336 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:22.414339 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:22.414341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:22.414536 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:43:22.414659 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 12:43:22.415305 (+ 646us) server_negotiation.cc:299] Negotiation successful
0927 12:43:22.415856 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":98,"threads_started":1}
Sep 27 12:43:22 dist-test-slave-bnc1 krb5kdc[13701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513001, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 12:43:22.415349 13863 master_runner.cc:186] Error getting master registration for 127.0.98.62:45903: OK, instance_id { permanent_uuid: "4d3f3a636fe040cabc19b7695316a42b" instance_seqno: 1790513001939822 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 12:43:22.419509 13863 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 12:43:22.421416 13785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:22.406150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44311 (local address 127.0.0.1:46294)
0927 12:43:22.406305 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:22.406317 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 12:43:22.406392 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:43:22.412477 (+ 6085us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:43:22.412481 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:43:22.412679 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:43:22.412778 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:22.412789 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:22.413711 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:22.413714 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:22.414471 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:22.414482 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:22.415102 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:22.415106 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:22.415204 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:22.415639 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:43:22.415657 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:43:22.417157 (+ 1500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:43:22.419843 (+ 2686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:43:22.419846 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:43:22.419848 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:43:22.420096 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:43:22.420423 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:43:22.420426 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:43:22.420428 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:43:22.420522 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:43:22.420965 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:43:22.420971 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:43:22.421061 (+ 90us) client_negotiation.cc:769] Sending connection context
0927 12:43:22.421155 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 12:43:22.421270 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"mutex_wait_us":3}
I20260927 12:43:22.421917 13935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:22.412065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46294 (local address 127.0.98.60:44311)
0927 12:43:22.412176 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 12:43:22.412180 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:43:22.412193 (+ 13us) server_negotiation.cc:407] Connection header received
0927 12:43:22.412224 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:43:22.412226 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:43:22.412261 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:43:22.412352 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:43:22.412904 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:22.413545 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:22.414610 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:22.414907 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:22.415001 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:22.418073 (+ 3072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:43:22.418077 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:43:22.418079 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:43:22.418103 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:43:22.419717 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:22.420227 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:22.420230 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:22.420232 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:22.420327 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:22.420617 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:22.420622 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:22.420625 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:22.420817 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:43:22.420908 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 12:43:22.421676 (+ 768us) server_negotiation.cc:299] Negotiation successful
0927 12:43:22.421781 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 12:43:22.422260 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:22.402888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58976 (local address 127.0.98.62:45903)
0927 12:43:22.403003 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 12:43:22.403008 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:43:22.411651 (+ 8643us) server_negotiation.cc:407] Connection header received
0927 12:43:22.411703 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:43:22.411707 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:43:22.411764 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:43:22.411853 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:43:22.412822 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:22.413634 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:22.414686 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:22.414999 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:22.415091 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:22.416729 (+ 1638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:43:22.416734 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:43:22.416737 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:43:22.416764 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:43:22.418772 (+ 2008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:22.419689 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:22.419693 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:22.419694 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:22.419778 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:22.420131 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:22.420136 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:22.420138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:22.420352 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:43:22.420481 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0927 12:43:22.421281 (+ 800us) server_negotiation.cc:299] Negotiation successful
0927 12:43:22.421890 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":381,"server-negotiator.queue_time_us":49}
I20260927 12:43:22.422895 13857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04c92d78793443b28b58aac89e6e65c1: No bootstrap required, opened a new log
I20260927 12:43:22.424489 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:22.402257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59360 (local address 127.0.98.61:42127)
0927 12:43:22.402374 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 12:43:22.402378 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:43:22.403715 (+ 1337us) server_negotiation.cc:407] Connection header received
0927 12:43:22.403848 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:43:22.403851 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:43:22.403887 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:43:22.403976 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:43:22.405183 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:22.405894 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:22.418016 (+ 12122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:22.418392 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:22.418477 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:22.420531 (+ 2054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:43:22.420534 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:43:22.420537 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:43:22.420560 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:43:22.422205 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:22.423208 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:22.423213 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:22.423216 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:22.423301 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:22.423604 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:22.423610 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:22.423613 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:22.423819 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:43:22.423940 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 12:43:22.424170 (+ 230us) server_negotiation.cc:299] Negotiation successful
0927 12:43:22.424292 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 12:43:22.424959 13782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d3f3a636fe040cabc19b7695316a42b: Bootstrap starting.
W20260927 12:43:22.427590 13958 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 37f12cb097cd4887b3ba4da5921f5197: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260927 12:43:22.427634 13782 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d3f3a636fe040cabc19b7695316a42b: Neither blocks nor log segments found. Creating new log.
W20260927 12:43:22.427649 13958 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 12:43:22.428478 13782 log.cc:824] T 00000000000000000000000000000000 P 4d3f3a636fe040cabc19b7695316a42b: Log is configured to *not* fsync() on all Append() calls
I20260927 12:43:22.429280 13857 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04c92d78793443b28b58aac89e6e65c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f3a636fe040cabc19b7695316a42b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45903 } } peers { permanent_uuid: "04c92d78793443b28b58aac89e6e65c1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42127 } } peers { permanent_uuid: "37f12cb097cd4887b3ba4da5921f5197" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44311 } }
I20260927 12:43:22.429497 13857 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04c92d78793443b28b58aac89e6e65c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:43:22.429559 13857 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04c92d78793443b28b58aac89e6e65c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04c92d78793443b28b58aac89e6e65c1, State: Initialized, Role: FOLLOWER
I20260927 12:43:22.430027 13857 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04c92d78793443b28b58aac89e6e65c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f3a636fe040cabc19b7695316a42b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45903 } } peers { permanent_uuid: "04c92d78793443b28b58aac89e6e65c1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42127 } } peers { permanent_uuid: "37f12cb097cd4887b3ba4da5921f5197" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44311 } }
I20260927 12:43:22.430493 13961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04c92d78793443b28b58aac89e6e65c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f3a636fe040cabc19b7695316a42b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45903 } } peers { permanent_uuid: "04c92d78793443b28b58aac89e6e65c1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42127 } } peers { permanent_uuid: "37f12cb097cd4887b3ba4da5921f5197" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44311 } } }
I20260927 12:43:22.430685 13961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04c92d78793443b28b58aac89e6e65c1 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 12:43:22.431303 13857 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04c92d78793443b28b58aac89e6e65c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:43:22.431352 13831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "37f12cb097cd4887b3ba4da5921f5197" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04c92d78793443b28b58aac89e6e65c1" is_pre_election: true
I20260927 12:43:22.431598 13831 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 04c92d78793443b28b58aac89e6e65c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37f12cb097cd4887b3ba4da5921f5197 in term 0.
I20260927 12:43:22.432062 13880 leader_election.cc:302] T 00000000000000000000000000000000 P 37f12cb097cd4887b3ba4da5921f5197 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04c92d78793443b28b58aac89e6e65c1, 37f12cb097cd4887b3ba4da5921f5197; no voters:
I20260927 12:43:22.432351 13946 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37f12cb097cd4887b3ba4da5921f5197 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:43:22.432422 13946 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37f12cb097cd4887b3ba4da5921f5197 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:43:22.432469 13946 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37f12cb097cd4887b3ba4da5921f5197 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:43:22.433768 13946 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37f12cb097cd4887b3ba4da5921f5197 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f3a636fe040cabc19b7695316a42b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45903 } } peers { permanent_uuid: "04c92d78793443b28b58aac89e6e65c1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42127 } } peers { permanent_uuid: "37f12cb097cd4887b3ba4da5921f5197" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44311 } }
I20260927 12:43:22.434118 13946 leader_election.cc:288] T 00000000000000000000000000000000 P 37f12cb097cd4887b3ba4da5921f5197 [CANDIDATE]: Term 1 election: Requested vote from peers 4d3f3a636fe040cabc19b7695316a42b (127.0.98.62:45903), 04c92d78793443b28b58aac89e6e65c1 (127.0.98.61:42127)
I20260927 12:43:22.435065 13831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "37f12cb097cd4887b3ba4da5921f5197" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04c92d78793443b28b58aac89e6e65c1"
I20260927 12:43:22.435206 13831 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04c92d78793443b28b58aac89e6e65c1 [term 0 FOLLOWER]: Advancing to term 1
W20260927 12:43:22.436164 13972 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 04c92d78793443b28b58aac89e6e65c1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 12:43:22.436223 13972 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 12:43:22.436472 13831 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 04c92d78793443b28b58aac89e6e65c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37f12cb097cd4887b3ba4da5921f5197 in term 1.
I20260927 12:43:22.436920 13880 leader_election.cc:302] T 00000000000000000000000000000000 P 37f12cb097cd4887b3ba4da5921f5197 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04c92d78793443b28b58aac89e6e65c1, 37f12cb097cd4887b3ba4da5921f5197; no voters:
I20260927 12:43:22.437196 13946 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37f12cb097cd4887b3ba4da5921f5197 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:43:22.437404 13946 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37f12cb097cd4887b3ba4da5921f5197 [term 1 LEADER]: Becoming Leader. State: Replica: 37f12cb097cd4887b3ba4da5921f5197, State: Running, Role: LEADER
I20260927 12:43:22.437824 13946 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37f12cb097cd4887b3ba4da5921f5197 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f3a636fe040cabc19b7695316a42b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45903 } } peers { permanent_uuid: "04c92d78793443b28b58aac89e6e65c1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42127 } } peers { permanent_uuid: "37f12cb097cd4887b3ba4da5921f5197" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44311 } }
I20260927 12:43:22.434566 13789 master_runner.cc:186] Error getting master registration for 127.0.98.62:45903: OK, instance_id { permanent_uuid: "4d3f3a636fe040cabc19b7695316a42b" instance_seqno: 1790513001939822 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 12:43:22.439266 13789 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 12:43:22.441166 13782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d3f3a636fe040cabc19b7695316a42b: No bootstrap required, opened a new log
I20260927 12:43:22.444578 13974 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37f12cb097cd4887b3ba4da5921f5197 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37f12cb097cd4887b3ba4da5921f5197. Latest consensus state: current_term: 1 leader_uuid: "37f12cb097cd4887b3ba4da5921f5197" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f3a636fe040cabc19b7695316a42b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45903 } } peers { permanent_uuid: "04c92d78793443b28b58aac89e6e65c1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42127 } } peers { permanent_uuid: "37f12cb097cd4887b3ba4da5921f5197" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44311 } } }
I20260927 12:43:22.444702 13974 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37f12cb097cd4887b3ba4da5921f5197 [sys.catalog]: This master's current role is: LEADER
I20260927 12:43:22.445232 13782 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d3f3a636fe040cabc19b7695316a42b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f3a636fe040cabc19b7695316a42b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45903 } } peers { permanent_uuid: "04c92d78793443b28b58aac89e6e65c1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42127 } } peers { permanent_uuid: "37f12cb097cd4887b3ba4da5921f5197" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44311 } }
I20260927 12:43:22.445469 13782 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d3f3a636fe040cabc19b7695316a42b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:43:22.445529 13782 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d3f3a636fe040cabc19b7695316a42b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3f3a636fe040cabc19b7695316a42b, State: Initialized, Role: FOLLOWER
I20260927 12:43:22.446005 13782 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d3f3a636fe040cabc19b7695316a42b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f3a636fe040cabc19b7695316a42b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45903 } } peers { permanent_uuid: "04c92d78793443b28b58aac89e6e65c1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42127 } } peers { permanent_uuid: "37f12cb097cd4887b3ba4da5921f5197" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44311 } }
I20260927 12:43:22.446539 13977 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d3f3a636fe040cabc19b7695316a42b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f3a636fe040cabc19b7695316a42b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45903 } } peers { permanent_uuid: "04c92d78793443b28b58aac89e6e65c1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42127 } } peers { permanent_uuid: "37f12cb097cd4887b3ba4da5921f5197" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44311 } } }
I20260927 12:43:22.446794 13977 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d3f3a636fe040cabc19b7695316a42b [sys.catalog]: This master's current role is: FOLLOWER
I20260927 12:43:22.447348 13782 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d3f3a636fe040cabc19b7695316a42b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:43:22.447383 13755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "37f12cb097cd4887b3ba4da5921f5197" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3f3a636fe040cabc19b7695316a42b"
I20260927 12:43:22.447578 13755 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d3f3a636fe040cabc19b7695316a42b [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:43:22.448853 13755 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4d3f3a636fe040cabc19b7695316a42b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37f12cb097cd4887b3ba4da5921f5197 in term 1.
I20260927 12:43:22.449641 13756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "37f12cb097cd4887b3ba4da5921f5197" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3f3a636fe040cabc19b7695316a42b" is_pre_election: true
I20260927 12:43:22.450019 13756 raft_consensus.cc:2362] T 00000000000000000000000000000000 P 4d3f3a636fe040cabc19b7695316a42b [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 37f12cb097cd4887b3ba4da5921f5197 in term 1. Re-sending same reply.
I20260927 12:43:22.453836 13976 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:43:22.454674 13976 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260927 12:43:22.458761 13988 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4d3f3a636fe040cabc19b7695316a42b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 12:43:22.458925 13988 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 12:43:22.460835 13756 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4d3f3a636fe040cabc19b7695316a42b [term 1 FOLLOWER]: Refusing update from remote peer 37f12cb097cd4887b3ba4da5921f5197: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:43:22.460888 13831 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 04c92d78793443b28b58aac89e6e65c1 [term 1 FOLLOWER]: Refusing update from remote peer 37f12cb097cd4887b3ba4da5921f5197: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:43:22.461434 13946 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 37f12cb097cd4887b3ba4da5921f5197 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04c92d78793443b28b58aac89e6e65c1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:43:22.461602 13974 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 37f12cb097cd4887b3ba4da5921f5197 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3f3a636fe040cabc19b7695316a42b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:43:22.464828 13977 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d3f3a636fe040cabc19b7695316a42b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37f12cb097cd4887b3ba4da5921f5197. Latest consensus state: current_term: 1 leader_uuid: "37f12cb097cd4887b3ba4da5921f5197" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f3a636fe040cabc19b7695316a42b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45903 } } peers { permanent_uuid: "04c92d78793443b28b58aac89e6e65c1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42127 } } peers { permanent_uuid: "37f12cb097cd4887b3ba4da5921f5197" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44311 } } }
I20260927 12:43:22.464937 13977 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d3f3a636fe040cabc19b7695316a42b [sys.catalog]: This master's current role is: FOLLOWER
I20260927 12:43:22.465059 13961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04c92d78793443b28b58aac89e6e65c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37f12cb097cd4887b3ba4da5921f5197. Latest consensus state: current_term: 1 leader_uuid: "37f12cb097cd4887b3ba4da5921f5197" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f3a636fe040cabc19b7695316a42b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45903 } } peers { permanent_uuid: "04c92d78793443b28b58aac89e6e65c1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42127 } } peers { permanent_uuid: "37f12cb097cd4887b3ba4da5921f5197" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44311 } } }
I20260927 12:43:22.465152 13961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04c92d78793443b28b58aac89e6e65c1 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 12:43:22.471627 13946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37f12cb097cd4887b3ba4da5921f5197 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "37f12cb097cd4887b3ba4da5921f5197" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f3a636fe040cabc19b7695316a42b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45903 } } peers { permanent_uuid: "04c92d78793443b28b58aac89e6e65c1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42127 } } peers { permanent_uuid: "37f12cb097cd4887b3ba4da5921f5197" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44311 } } }
I20260927 12:43:22.471752 13946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37f12cb097cd4887b3ba4da5921f5197 [sys.catalog]: This master's current role is: LEADER
I20260927 12:43:22.472026 13961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04c92d78793443b28b58aac89e6e65c1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "37f12cb097cd4887b3ba4da5921f5197" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f3a636fe040cabc19b7695316a42b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45903 } } peers { permanent_uuid: "04c92d78793443b28b58aac89e6e65c1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42127 } } peers { permanent_uuid: "37f12cb097cd4887b3ba4da5921f5197" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44311 } } }
I20260927 12:43:22.472136 13961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04c92d78793443b28b58aac89e6e65c1 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 12:43:22.472071 13946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37f12cb097cd4887b3ba4da5921f5197 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "37f12cb097cd4887b3ba4da5921f5197" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f3a636fe040cabc19b7695316a42b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45903 } } peers { permanent_uuid: "04c92d78793443b28b58aac89e6e65c1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42127 } } peers { permanent_uuid: "37f12cb097cd4887b3ba4da5921f5197" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44311 } } }
I20260927 12:43:22.472175 13946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37f12cb097cd4887b3ba4da5921f5197 [sys.catalog]: This master's current role is: LEADER
I20260927 12:43:22.472303 13992 mvcc.cc:205] Tried to move back new op lower bound from 7333941258073796608 to 7333941257989197824. Current Snapshot: MvccSnapshot[applied={T|T < 7333941258073796608}]
I20260927 12:43:22.472411 13989 mvcc.cc:205] Tried to move back new op lower bound from 7333941258073796608 to 7333941257989197824. Current Snapshot: MvccSnapshot[applied={T|T < 7333941258073796608}]
I20260927 12:43:22.473058 13977 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d3f3a636fe040cabc19b7695316a42b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "37f12cb097cd4887b3ba4da5921f5197" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3f3a636fe040cabc19b7695316a42b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45903 } } peers { permanent_uuid: "04c92d78793443b28b58aac89e6e65c1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42127 } } peers { permanent_uuid: "37f12cb097cd4887b3ba4da5921f5197" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44311 } } }
I20260927 12:43:22.473163 13977 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d3f3a636fe040cabc19b7695316a42b [sys.catalog]: This master's current role is: FOLLOWER
I20260927 12:43:22.474872 13991 mvcc.cc:205] Tried to move back new op lower bound from 7333941258073796608 to 7333941257989197824. Current Snapshot: MvccSnapshot[applied={T|T < 7333941258073796608}]
I20260927 12:43:22.482721 13976 catalog_manager.cc:1347] Generated new cluster ID: 39af8506f4174e5db4f26dba65accddd
I20260927 12:43:22.482791 13976 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:43:22.521639 13976 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:43:22.522979 13976 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:43:22.534726 13976 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 37f12cb097cd4887b3ba4da5921f5197: Generated new TSK 0
I20260927 12:43:22.535533 13976 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:43:23.437467 13972 catalog_manager.cc:1259] Loaded cluster ID: 39af8506f4174e5db4f26dba65accddd
I20260927 12:43:23.437577 13972 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 04c92d78793443b28b58aac89e6e65c1: loading cluster ID for follower catalog manager: success
I20260927 12:43:23.439254 13972 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 04c92d78793443b28b58aac89e6e65c1: acquiring CA information for follower catalog manager: success
I20260927 12:43:23.440459 13972 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 04c92d78793443b28b58aac89e6e65c1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 12:43:23.460292 13988 catalog_manager.cc:1259] Loaded cluster ID: 39af8506f4174e5db4f26dba65accddd
I20260927 12:43:23.460376 13988 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4d3f3a636fe040cabc19b7695316a42b: loading cluster ID for follower catalog manager: success
I20260927 12:43:23.462143 13988 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4d3f3a636fe040cabc19b7695316a42b: acquiring CA information for follower catalog manager: success
I20260927 12:43:23.463320 13988 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4d3f3a636fe040cabc19b7695316a42b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 12:43:23.596171 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 13714
I20260927 12:43:23.603137 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 13789
I20260927 12:43:23.609154 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 13863
2026-09-27T12:43:23Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3026 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790512840836255-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:43:23 dist-test-slave-bnc1 krb5kdc[14005](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:43:23 dist-test-slave-bnc1 krb5kdc[14005](info): set up 2 sockets
Sep 27 12:43:23 dist-test-slave-bnc1 krb5kdc[14005](info): commencing operation
krb5kdc: starting...
W20260927 12:43:24.657830 392 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790512840836255-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790512840836255-392-0/krb5kdc/test-user.keytab.
Sep 27 12:43:24 dist-test-slave-bnc1 krb5kdc[14005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513004, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:43:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:43:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:43:24.780064 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790512840836255-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790512840836255-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790512840836255-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39481
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39481
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:43:24.887579 14018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:43:24.887833 14018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:43:24.887873 14018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:43:24.891831 14018 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:43:24.891888 14018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:43:24.891908 14018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:43:24.891924 14018 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:43:24.891939 14018 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:43:24.897078 14018 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790512840836255-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790512840836255-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39481
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14018
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790512840836255-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:43:24.898156 14018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:43:24.899099 14018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:43:24.907625 14026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:43:24.907625 14024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:43:24.907750 14018 server_base.cc:1034] running on GCE node
W20260927 12:43:24.907672 14023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:43:24.908540 14018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:43:24.909809 14018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:43:24.910975 14018 hybrid_clock.cc:648] HybridClock initialized: now 1790513004910934 us; error 52 us; skew 500 ppm
Sep 27 12:43:24 dist-test-slave-bnc1 krb5kdc[14005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513004, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:43:24.913754 14018 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:43:24.914634 14018 webserver.cc:460] Webserver started at http://127.0.98.62:41087/ using document root <none> and password file <none>
I20260927 12:43:24.915131 14018 fs_manager.cc:359] Metadata directory not provided
I20260927 12:43:24.915190 14018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:43:24.915409 14018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:43:24.917474 14018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790512840836255-392-0/minicluster-data/master-0/data/instance:
uuid: "1af4e664b2d94e7283b5aba95b9d1101"
format_stamp: "Formatted at 2026-09-27 12:43:24 on dist-test-slave-bnc1"
server_key: "6973eaba3e88e1fdeee6f8d1fefe1a89"
server_key_iv: "4272c4056eac807696c6dbc78e4da7a1"
server_key_version: "encryptionkey@0"
I20260927 12:43:24.918095 14018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790512840836255-392-0/minicluster-data/master-0/wal/instance:
uuid: "1af4e664b2d94e7283b5aba95b9d1101"
format_stamp: "Formatted at 2026-09-27 12:43:24 on dist-test-slave-bnc1"
server_key: "6973eaba3e88e1fdeee6f8d1fefe1a89"
server_key_iv: "4272c4056eac807696c6dbc78e4da7a1"
server_key_version: "encryptionkey@0"
I20260927 12:43:24.921772 14018 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 12:43:24.924093 14033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:43:24.925285 14018 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:43:24.925419 14018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790512840836255-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790512840836255-392-0/minicluster-data/master-0/wal
uuid: "1af4e664b2d94e7283b5aba95b9d1101"
format_stamp: "Formatted at 2026-09-27 12:43:24 on dist-test-slave-bnc1"
server_key: "6973eaba3e88e1fdeee6f8d1fefe1a89"
server_key_iv: "4272c4056eac807696c6dbc78e4da7a1"
server_key_version: "encryptionkey@0"
I20260927 12:43:24.925540 14018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790512840836255-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790512840836255-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790512840836255-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:43:24.942775 14018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:43:24.946537 14018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:43:24.946754 14018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:43:24.953231 14018 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39481
I20260927 12:43:24.953320 14085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39481 every 8 connection(s)
I20260927 12:43:24.954241 14018 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:43:24.955993 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 14018
I20260927 12:43:24.956130 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790512840836255-392-0/minicluster-data/master-0/wal/instance
I20260927 12:43:24.956424 392 external_mini_cluster.cc:1428] Setting key 4359c09014a2cbd7c4ccd2fbd4d430a3
I20260927 12:43:24.957273 14086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:43:24.963627 14086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1af4e664b2d94e7283b5aba95b9d1101: Bootstrap starting.
Sep 27 12:43:24 dist-test-slave-bnc1 krb5kdc[14005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513004, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:43:24.966495 14086 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1af4e664b2d94e7283b5aba95b9d1101: Neither blocks nor log segments found. Creating new log.
I20260927 12:43:24.967334 14086 log.cc:824] T 00000000000000000000000000000000 P 1af4e664b2d94e7283b5aba95b9d1101: Log is configured to *not* fsync() on all Append() calls
I20260927 12:43:24.969398 14086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1af4e664b2d94e7283b5aba95b9d1101: No bootstrap required, opened a new log
I20260927 12:43:24.971172 14089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:24.957813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48106 (local address 127.0.98.62:39481)
0927 12:43:24.958215 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0927 12:43:24.958231 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 12:43:24.958267 (+ 36us) server_negotiation.cc:407] Connection header received
0927 12:43:24.958889 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:43:24.958904 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:43:24.959232 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:43:24.959515 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:43:24.960463 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:24.961490 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:24.962715 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:24.963207 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:24.963329 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:24.966240 (+ 2911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:43:24.966245 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:43:24.966257 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:43:24.966283 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:43:24.968119 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:24.968552 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:24.968557 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:24.968561 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:24.968665 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:24.968953 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:24.968956 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:24.968957 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:24.969355 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:43:24.969541 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0927 12:43:24.969874 (+ 333us) server_negotiation.cc:299] Negotiation successful
0927 12:43:24.970188 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":140,"threads_started":1}
I20260927 12:43:24.972656 14086 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1af4e664b2d94e7283b5aba95b9d1101 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1af4e664b2d94e7283b5aba95b9d1101" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39481 } }
I20260927 12:43:24.972908 14086 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1af4e664b2d94e7283b5aba95b9d1101 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:43:24.972996 14086 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1af4e664b2d94e7283b5aba95b9d1101 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1af4e664b2d94e7283b5aba95b9d1101, State: Initialized, Role: FOLLOWER
I20260927 12:43:24.973400 14086 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1af4e664b2d94e7283b5aba95b9d1101 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1af4e664b2d94e7283b5aba95b9d1101" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39481 } }
I20260927 12:43:24.973520 14086 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1af4e664b2d94e7283b5aba95b9d1101 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:43:24.973631 14086 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1af4e664b2d94e7283b5aba95b9d1101 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:43:24.973734 14086 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1af4e664b2d94e7283b5aba95b9d1101 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:43:24.974436 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 12:43:24.974687 14086 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1af4e664b2d94e7283b5aba95b9d1101 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1af4e664b2d94e7283b5aba95b9d1101" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39481 } }
I20260927 12:43:24.974999 14086 leader_election.cc:302] T 00000000000000000000000000000000 P 1af4e664b2d94e7283b5aba95b9d1101 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1af4e664b2d94e7283b5aba95b9d1101; no voters:
I20260927 12:43:24.975366 14086 leader_election.cc:288] T 00000000000000000000000000000000 P 1af4e664b2d94e7283b5aba95b9d1101 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:43:24.975430 14091 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1af4e664b2d94e7283b5aba95b9d1101 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:43:24.975641 14091 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1af4e664b2d94e7283b5aba95b9d1101 [term 1 LEADER]: Becoming Leader. State: Replica: 1af4e664b2d94e7283b5aba95b9d1101, State: Running, Role: LEADER
I20260927 12:43:24.975921 14091 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1af4e664b2d94e7283b5aba95b9d1101 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1af4e664b2d94e7283b5aba95b9d1101" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39481 } }
I20260927 12:43:24.976557 14086 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1af4e664b2d94e7283b5aba95b9d1101 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:43:24.976984 14091 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1af4e664b2d94e7283b5aba95b9d1101 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1af4e664b2d94e7283b5aba95b9d1101. Latest consensus state: current_term: 1 leader_uuid: "1af4e664b2d94e7283b5aba95b9d1101" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1af4e664b2d94e7283b5aba95b9d1101" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39481 } } }
I20260927 12:43:24.977089 14091 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1af4e664b2d94e7283b5aba95b9d1101 [sys.catalog]: This master's current role is: LEADER
I20260927 12:43:24.977500 14092 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1af4e664b2d94e7283b5aba95b9d1101 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1af4e664b2d94e7283b5aba95b9d1101" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1af4e664b2d94e7283b5aba95b9d1101" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39481 } } }
I20260927 12:43:24.977662 14092 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1af4e664b2d94e7283b5aba95b9d1101 [sys.catalog]: This master's current role is: LEADER
I20260927 12:43:24.978943 14095 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:43:24.981359 14095 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:43:24.986425 14095 catalog_manager.cc:1347] Generated new cluster ID: 4d759102321042d88a01ef6eaae0d0e0
I20260927 12:43:24.986491 14095 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:43:25.016091 14095 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:43:25.017098 14095 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:43:25.022635 14095 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1af4e664b2d94e7283b5aba95b9d1101: Generated new TSK 0
I20260927 12:43:25.023324 14095 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:43:25.030606 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 14018
2026-09-27T12:43:25Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1419 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:43:25 dist-test-slave-bnc1 krb5kdc[14113](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:43:25 dist-test-slave-bnc1 krb5kdc[14113](info): set up 2 sockets
Sep 27 12:43:25 dist-test-slave-bnc1 krb5kdc[14113](info): commencing operation
krb5kdc: starting...
W20260927 12:43:26.053812 392 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/krb5kdc/test-user.keytab.
Sep 27 12:43:26 dist-test-slave-bnc1 krb5kdc[14113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:43:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:43:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:43:26.172487 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36945
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36945
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:43:26.280241 14126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:43:26.280534 14126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:43:26.280614 14126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:43:26.284587 14126 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:43:26.284680 14126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:43:26.284718 14126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:43:26.284755 14126 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:43:26.284785 14126 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:43:26.289909 14126 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36945
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:43:26.290994 14126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:43:26.292053 14126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:43:26.300312 14132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:43:26.300345 14131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:43:26.300329 14134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:43:26.300730 14126 server_base.cc:1034] running on GCE node
I20260927 12:43:26.301255 14126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:43:26.302408 14126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:43:26.303599 14126 hybrid_clock.cc:648] HybridClock initialized: now 1790513006303587 us; error 36 us; skew 500 ppm
Sep 27 12:43:26 dist-test-slave-bnc1 krb5kdc[14113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513006, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:43:26.306219 14126 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:43:26.307029 14126 webserver.cc:460] Webserver started at http://127.0.98.62:45379/ using document root <none> and password file <none>
I20260927 12:43:26.307581 14126 fs_manager.cc:359] Metadata directory not provided
I20260927 12:43:26.307672 14126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:43:26.307896 14126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:43:26.309947 14126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/master-0/data/instance:
uuid: "4027986ce39f4ac898f9f1f7b7a98a93"
format_stamp: "Formatted at 2026-09-27 12:43:26 on dist-test-slave-bnc1"
server_key: "8d9e049b3635deaa906bc249e97dca15"
server_key_iv: "f6360a5d8bbd11b7e31d6a6f64f31fd6"
server_key_version: "encryptionkey@0"
I20260927 12:43:26.310489 14126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/master-0/wal/instance:
uuid: "4027986ce39f4ac898f9f1f7b7a98a93"
format_stamp: "Formatted at 2026-09-27 12:43:26 on dist-test-slave-bnc1"
server_key: "8d9e049b3635deaa906bc249e97dca15"
server_key_iv: "f6360a5d8bbd11b7e31d6a6f64f31fd6"
server_key_version: "encryptionkey@0"
I20260927 12:43:26.314023 14126 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:43:26.316294 14141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:43:26.317415 14126 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:43:26.317553 14126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/master-0/wal
uuid: "4027986ce39f4ac898f9f1f7b7a98a93"
format_stamp: "Formatted at 2026-09-27 12:43:26 on dist-test-slave-bnc1"
server_key: "8d9e049b3635deaa906bc249e97dca15"
server_key_iv: "f6360a5d8bbd11b7e31d6a6f64f31fd6"
server_key_version: "encryptionkey@0"
I20260927 12:43:26.317735 14126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:43:26.327011 14126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:43:26.329764 14126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:43:26.329980 14126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:43:26.336820 14126 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36945
I20260927 12:43:26.337150 14193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36945 every 8 connection(s)
I20260927 12:43:26.337785 14126 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:43:26.338171 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 14126
I20260927 12:43:26.338320 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/master-0/wal/instance
I20260927 12:43:26.338604 392 external_mini_cluster.cc:1428] Setting key a7b42eb11c1ff480ba41e863c357e03f
I20260927 12:43:26.340830 14194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:43:26.347250 14194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4027986ce39f4ac898f9f1f7b7a98a93: Bootstrap starting.
Sep 27 12:43:26 dist-test-slave-bnc1 krb5kdc[14113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:43:26.349758 14194 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4027986ce39f4ac898f9f1f7b7a98a93: Neither blocks nor log segments found. Creating new log.
I20260927 12:43:26.350452 14194 log.cc:824] T 00000000000000000000000000000000 P 4027986ce39f4ac898f9f1f7b7a98a93: Log is configured to *not* fsync() on all Append() calls
I20260927 12:43:26.352380 14194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4027986ce39f4ac898f9f1f7b7a98a93: No bootstrap required, opened a new log
I20260927 12:43:26.352612 14197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:26.339826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45306 (local address 127.0.98.62:36945)
0927 12:43:26.340209 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0927 12:43:26.340224 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 12:43:26.340257 (+ 33us) server_negotiation.cc:407] Connection header received
0927 12:43:26.340926 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:43:26.340955 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:43:26.341266 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:43:26.341540 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:43:26.342453 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:26.343436 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:26.344571 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:26.345015 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:26.345127 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:26.348029 (+ 2902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:43:26.348034 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:43:26.348045 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:43:26.348070 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:43:26.349787 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:26.350235 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:26.350242 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:26.350247 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:26.350363 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:26.350702 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:26.350706 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:26.350708 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:26.351091 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:43:26.351297 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0927 12:43:26.351548 (+ 251us) server_negotiation.cc:299] Negotiation successful
0927 12:43:26.351842 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":132,"threads_started":1}
I20260927 12:43:26.356271 14194 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4027986ce39f4ac898f9f1f7b7a98a93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4027986ce39f4ac898f9f1f7b7a98a93" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36945 } }
I20260927 12:43:26.356518 14194 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4027986ce39f4ac898f9f1f7b7a98a93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:43:26.356598 14194 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4027986ce39f4ac898f9f1f7b7a98a93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4027986ce39f4ac898f9f1f7b7a98a93, State: Initialized, Role: FOLLOWER
I20260927 12:43:26.357008 14194 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4027986ce39f4ac898f9f1f7b7a98a93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4027986ce39f4ac898f9f1f7b7a98a93" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36945 } }
I20260927 12:43:26.357152 14194 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4027986ce39f4ac898f9f1f7b7a98a93 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:43:26.357239 14194 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4027986ce39f4ac898f9f1f7b7a98a93 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:43:26.357360 14194 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4027986ce39f4ac898f9f1f7b7a98a93 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:43:26.358296 14194 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4027986ce39f4ac898f9f1f7b7a98a93 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4027986ce39f4ac898f9f1f7b7a98a93" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36945 } }
I20260927 12:43:26.358629 14194 leader_election.cc:302] T 00000000000000000000000000000000 P 4027986ce39f4ac898f9f1f7b7a98a93 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4027986ce39f4ac898f9f1f7b7a98a93; no voters:
I20260927 12:43:26.358942 14194 leader_election.cc:288] T 00000000000000000000000000000000 P 4027986ce39f4ac898f9f1f7b7a98a93 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:43:26.359066 14199 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4027986ce39f4ac898f9f1f7b7a98a93 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:43:26.359336 14199 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4027986ce39f4ac898f9f1f7b7a98a93 [term 1 LEADER]: Becoming Leader. State: Replica: 4027986ce39f4ac898f9f1f7b7a98a93, State: Running, Role: LEADER
I20260927 12:43:26.359695 14199 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4027986ce39f4ac898f9f1f7b7a98a93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4027986ce39f4ac898f9f1f7b7a98a93" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36945 } }
I20260927 12:43:26.360174 14194 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4027986ce39f4ac898f9f1f7b7a98a93 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:43:26.361299 14200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4027986ce39f4ac898f9f1f7b7a98a93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4027986ce39f4ac898f9f1f7b7a98a93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4027986ce39f4ac898f9f1f7b7a98a93" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36945 } } }
I20260927 12:43:26.361438 14200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4027986ce39f4ac898f9f1f7b7a98a93 [sys.catalog]: This master's current role is: LEADER
I20260927 12:43:26.361634 14201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4027986ce39f4ac898f9f1f7b7a98a93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4027986ce39f4ac898f9f1f7b7a98a93. Latest consensus state: current_term: 1 leader_uuid: "4027986ce39f4ac898f9f1f7b7a98a93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4027986ce39f4ac898f9f1f7b7a98a93" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36945 } } }
I20260927 12:43:26.361807 14201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4027986ce39f4ac898f9f1f7b7a98a93 [sys.catalog]: This master's current role is: LEADER
I20260927 12:43:26.362103 14209 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:43:26.365240 14209 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:43:26.370313 14209 catalog_manager.cc:1347] Generated new cluster ID: 102886e1aa324ab1bf5890f54a30055d
I20260927 12:43:26.370378 14209 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:43:26.386814 14209 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:43:26.387681 14209 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:43:26.402652 14209 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4027986ce39f4ac898f9f1f7b7a98a93: Generated new TSK 0
I20260927 12:43:26.403558 14209 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 12:43:26.440409 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36945
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:43:26.551400 14220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:43:26.551648 14220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:43:26.551694 14220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:43:26.555625 14220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:43:26.555675 14220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:43:26.555728 14220 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:43:26.560778 14220 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14220
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:43:26.561846 14220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:43:26.562832 14220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:43:26.572351 14226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:43:26.572360 14225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:43:26.572436 14228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:43:26.572420 14220 server_base.cc:1034] running on GCE node
I20260927 12:43:26.572969 14220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:43:26.573630 14220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:43:26.574796 14220 hybrid_clock.cc:648] HybridClock initialized: now 1790513006574780 us; error 28 us; skew 500 ppm
Sep 27 12:43:26 dist-test-slave-bnc1 krb5kdc[14113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513006, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:43:26.582144 14220 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 12:43:26.583014 14220 webserver.cc:460] Webserver started at http://127.0.98.1:43873/ using document root <none> and password file <none>
I20260927 12:43:26.583518 14220 fs_manager.cc:359] Metadata directory not provided
I20260927 12:43:26.583573 14220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:43:26.583729 14220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:43:26.585696 14220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-0/data/instance:
uuid: "3dc0e90d397a4c1bb3749bf9a7c50732"
format_stamp: "Formatted at 2026-09-27 12:43:26 on dist-test-slave-bnc1"
server_key: "4b83bf9be58a116b974810c085611cce"
server_key_iv: "3532482f32615bc41ad994ad71f535f9"
server_key_version: "encryptionkey@0"
I20260927 12:43:26.586242 14220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3dc0e90d397a4c1bb3749bf9a7c50732"
format_stamp: "Formatted at 2026-09-27 12:43:26 on dist-test-slave-bnc1"
server_key: "4b83bf9be58a116b974810c085611cce"
server_key_iv: "3532482f32615bc41ad994ad71f535f9"
server_key_version: "encryptionkey@0"
I20260927 12:43:26.589774 14220 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:43:26.592146 14235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:43:26.593326 14220 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:43:26.593446 14220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-0/wal
uuid: "3dc0e90d397a4c1bb3749bf9a7c50732"
format_stamp: "Formatted at 2026-09-27 12:43:26 on dist-test-slave-bnc1"
server_key: "4b83bf9be58a116b974810c085611cce"
server_key_iv: "3532482f32615bc41ad994ad71f535f9"
server_key_version: "encryptionkey@0"
I20260927 12:43:26.593550 14220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:43:26.614715 14220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:43:26.617825 14220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:43:26.618027 14220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:43:26.618619 14220 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:43:26.619436 14220 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:43:26.619484 14220 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:43:26.619565 14220 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:43:26.619596 14220 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:43:26.628213 14220 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38451
I20260927 12:43:26.628265 14348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38451 every 8 connection(s)
I20260927 12:43:26.629153 14220 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:43:26.636698 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 14220
I20260927 12:43:26.636855 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-0/wal/instance
I20260927 12:43:26.637213 392 external_mini_cluster.cc:1428] Setting key 61a995b1cfa03b41bd623aeaaf4b36e4
Sep 27 12:43:26 dist-test-slave-bnc1 krb5kdc[14113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513006, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:43:26.643126 14197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:26.630996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35889 (local address 127.0.98.62:36945)
0927 12:43:26.631168 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 12:43:26.631172 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:43:26.632194 (+ 1022us) server_negotiation.cc:407] Connection header received
0927 12:43:26.633042 (+ 848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:43:26.633044 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:43:26.633091 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:43:26.633167 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:43:26.634398 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:26.634937 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:26.636083 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:26.636300 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:26.636378 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:26.639230 (+ 2852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:43:26.639233 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:43:26.639235 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:43:26.639260 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:43:26.640709 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:26.641347 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:26.641369 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:26.641373 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:26.641467 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:26.641921 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:26.641924 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:26.641925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:26.642107 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:43:26.642188 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 12:43:26.642881 (+ 693us) server_negotiation.cc:299] Negotiation successful
0927 12:43:26.642995 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 12:43:26.644409 14351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:26.631393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36945 (local address 127.0.98.1:35889)
0927 12:43:26.632046 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:26.632089 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 12:43:26.632858 (+ 769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:43:26.633325 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:43:26.633331 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:43:26.633795 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:43:26.634272 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:26.634285 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:26.635068 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:26.635071 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:26.635978 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:26.635988 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:26.636471 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:26.636475 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:26.636642 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:26.637336 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:43:26.637352 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:43:26.639050 (+ 1698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:43:26.640865 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:43:26.640871 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:43:26.640887 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:43:26.641213 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:43:26.641600 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:43:26.641606 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:43:26.641611 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:43:26.641821 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:43:26.642218 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:43:26.642227 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:43:26.642578 (+ 351us) client_negotiation.cc:769] Sending connection context
0927 12:43:26.642807 (+ 229us) client_negotiation.cc:240] Negotiation successful
0927 12:43:26.643223 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":116,"threads_started":1}
I20260927 12:43:26.645812 14349 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36945
I20260927 12:43:26.646090 14349 heartbeater.cc:464] Registering TS with master...
I20260927 12:43:26.646608 14349 heartbeater.cc:511] Master 127.0.98.62:36945 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 12:43:26.648435 14158 ts_manager.cc:194] Registered new tserver with Master: 3dc0e90d397a4c1bb3749bf9a7c50732 (127.0.98.1:38451)
I20260927 12:43:26.649689 14158 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35889
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 12:43:26.665148 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36945
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:43:26.777652 14354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:43:26.777935 14354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:43:26.777980 14354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:43:26.782068 14354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:43:26.782130 14354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:43:26.782189 14354 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:43:26.787587 14354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14354
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:43:26.788626 14354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:43:26.789606 14354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 12:43:26.799724 14354 server_base.cc:1034] running on GCE node
W20260927 12:43:26.799680 14360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:43:26.799679 14359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:43:26.799676 14362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:43:26.800426 14354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:43:26.801218 14354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:43:26.802433 14354 hybrid_clock.cc:648] HybridClock initialized: now 1790513006802398 us; error 41 us; skew 500 ppm
Sep 27 12:43:26 dist-test-slave-bnc1 krb5kdc[14113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513006, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:43:26.805280 14354 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 12:43:26.806177 14354 webserver.cc:460] Webserver started at http://127.0.98.2:35803/ using document root <none> and password file <none>
I20260927 12:43:26.806728 14354 fs_manager.cc:359] Metadata directory not provided
I20260927 12:43:26.806816 14354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:43:26.807044 14354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:43:26.809033 14354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-1/data/instance:
uuid: "79e57a29f88f433aa4f1215486f9545d"
format_stamp: "Formatted at 2026-09-27 12:43:26 on dist-test-slave-bnc1"
server_key: "aa1e740c243c0d5bc534eca17e5e3c13"
server_key_iv: "1be7993cff0c56686ef49b7f9037e3e7"
server_key_version: "encryptionkey@0"
I20260927 12:43:26.809638 14354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-1/wal/instance:
uuid: "79e57a29f88f433aa4f1215486f9545d"
format_stamp: "Formatted at 2026-09-27 12:43:26 on dist-test-slave-bnc1"
server_key: "aa1e740c243c0d5bc534eca17e5e3c13"
server_key_iv: "1be7993cff0c56686ef49b7f9037e3e7"
server_key_version: "encryptionkey@0"
I20260927 12:43:26.813174 14354 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 12:43:26.815635 14369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:43:26.816855 14354 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:43:26.816990 14354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-1/wal
uuid: "79e57a29f88f433aa4f1215486f9545d"
format_stamp: "Formatted at 2026-09-27 12:43:26 on dist-test-slave-bnc1"
server_key: "aa1e740c243c0d5bc534eca17e5e3c13"
server_key_iv: "1be7993cff0c56686ef49b7f9037e3e7"
server_key_version: "encryptionkey@0"
I20260927 12:43:26.817097 14354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:43:26.832468 14354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:43:26.835729 14354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:43:26.835932 14354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:43:26.836508 14354 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:43:26.837323 14354 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:43:26.837402 14354 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:43:26.837496 14354 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:43:26.837546 14354 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:43:26.846619 14354 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33847
I20260927 12:43:26.846639 14482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33847 every 8 connection(s)
I20260927 12:43:26.847586 14354 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:43:26.851635 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 14354
I20260927 12:43:26.851784 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-1/wal/instance
I20260927 12:43:26.852115 392 external_mini_cluster.cc:1428] Setting key 80345e260e162771ef1ec68b54741639
Sep 27 12:43:26 dist-test-slave-bnc1 krb5kdc[14113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513006, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 12:43:26.862834 14197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:26.849432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44715 (local address 127.0.98.62:36945)
0927 12:43:26.849654 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 12:43:26.849665 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 12:43:26.850697 (+ 1032us) server_negotiation.cc:407] Connection header received
0927 12:43:26.851640 (+ 943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:43:26.851643 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:43:26.851703 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:43:26.851809 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:43:26.853255 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:26.853923 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:26.855685 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:26.856059 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:26.856172 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:26.858682 (+ 2510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:43:26.858688 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:43:26.858694 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:43:26.858729 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:43:26.860398 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:26.861141 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:26.861148 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:26.861152 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:26.861245 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:26.861620 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:26.861626 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:26.861630 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:26.861883 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:43:26.862000 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 12:43:26.862609 (+ 609us) server_negotiation.cc:299] Negotiation successful
0927 12:43:26.862715 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 12:43:26.863705 14485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:26.849829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36945 (local address 127.0.98.2:44715)
0927 12:43:26.850541 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:26.850584 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 12:43:26.851421 (+ 837us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:43:26.852078 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:43:26.852085 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:43:26.852510 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:43:26.853109 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:26.853124 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:26.854102 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:26.854108 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:26.855565 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:26.855576 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:26.856208 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:26.856211 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:26.856354 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:26.856976 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:43:26.856993 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:43:26.858490 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:43:26.860535 (+ 2045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:43:26.860544 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:43:26.860581 (+ 37us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:43:26.860976 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:43:26.861339 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:43:26.861341 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:43:26.861343 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:43:26.861487 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:43:26.862008 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:43:26.862045 (+ 37us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:43:26.862358 (+ 313us) client_negotiation.cc:769] Sending connection context
0927 12:43:26.862556 (+ 198us) client_negotiation.cc:240] Negotiation successful
0927 12:43:26.862870 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":117,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 12:43:26.864948 14483 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36945
I20260927 12:43:26.865250 14483 heartbeater.cc:464] Registering TS with master...
I20260927 12:43:26.865876 14483 heartbeater.cc:511] Master 127.0.98.62:36945 requested a full tablet report, sending...
I20260927 12:43:26.867039 14158 ts_manager.cc:194] Registered new tserver with Master: 79e57a29f88f433aa4f1215486f9545d (127.0.98.2:33847)
I20260927 12:43:26.867635 14158 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44715
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 12:43:26.878890 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36945
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:43:26.987771 14488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:43:26.988112 14488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:43:26.988158 14488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:43:26.992187 14488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:43:26.992249 14488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:43:26.992306 14488 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:43:26.997689 14488 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14488
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:43:26.998744 14488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:43:26.999708 14488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:43:27.009473 14496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:43:27.009459 14494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:43:27.009562 14488 server_base.cc:1034] running on GCE node
W20260927 12:43:27.009479 14493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:43:27.010067 14488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:43:27.010743 14488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:43:27.011914 14488 hybrid_clock.cc:648] HybridClock initialized: now 1790513007011897 us; error 28 us; skew 500 ppm
Sep 27 12:43:27 dist-test-slave-bnc1 krb5kdc[14113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513007, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:43:27.014837 14488 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 12:43:27.015693 14488 webserver.cc:460] Webserver started at http://127.0.98.3:44043/ using document root <none> and password file <none>
I20260927 12:43:27.016247 14488 fs_manager.cc:359] Metadata directory not provided
I20260927 12:43:27.016305 14488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:43:27.016546 14488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:43:27.018620 14488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-2/data/instance:
uuid: "02d683b804bd404aa1ea624a04611726"
format_stamp: "Formatted at 2026-09-27 12:43:27 on dist-test-slave-bnc1"
server_key: "990cf133240c5c145383d0b758b2e750"
server_key_iv: "2a7b5d2c81a9e9c7f980197f03a1ab35"
server_key_version: "encryptionkey@0"
I20260927 12:43:27.019199 14488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-2/wal/instance:
uuid: "02d683b804bd404aa1ea624a04611726"
format_stamp: "Formatted at 2026-09-27 12:43:27 on dist-test-slave-bnc1"
server_key: "990cf133240c5c145383d0b758b2e750"
server_key_iv: "2a7b5d2c81a9e9c7f980197f03a1ab35"
server_key_version: "encryptionkey@0"
I20260927 12:43:27.023737 14488 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 12:43:27.026108 14503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:43:27.027348 14488 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 12:43:27.027486 14488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-2/wal
uuid: "02d683b804bd404aa1ea624a04611726"
format_stamp: "Formatted at 2026-09-27 12:43:27 on dist-test-slave-bnc1"
server_key: "990cf133240c5c145383d0b758b2e750"
server_key_iv: "2a7b5d2c81a9e9c7f980197f03a1ab35"
server_key_version: "encryptionkey@0"
I20260927 12:43:27.027597 14488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:43:27.036880 14488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:43:27.040228 14488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:43:27.040436 14488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:43:27.041049 14488 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:43:27.041937 14488 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:43:27.042016 14488 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:43:27.042088 14488 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:43:27.042133 14488 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:43:27.050801 14488 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35185
I20260927 12:43:27.050827 14616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35185 every 8 connection(s)
I20260927 12:43:27.051692 14488 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:43:27.054808 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 14488
I20260927 12:43:27.054957 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-2/wal/instance
I20260927 12:43:27.055224 392 external_mini_cluster.cc:1428] Setting key b326db190e26763e79a9fa9d7298cd7a
Sep 27 12:43:27 dist-test-slave-bnc1 krb5kdc[14113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513007, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:43:27.065466 14197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:27.053610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50689 (local address 127.0.98.62:36945)
0927 12:43:27.053758 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 12:43:27.053762 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:43:27.054804 (+ 1042us) server_negotiation.cc:407] Connection header received
0927 12:43:27.055941 (+ 1137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:43:27.055945 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:43:27.055999 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:43:27.056121 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:43:27.057364 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:27.057948 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:27.059101 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:27.059306 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:27.059414 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:27.062112 (+ 2698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:43:27.062134 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:43:27.062139 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:43:27.062193 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:43:27.063468 (+ 1275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:27.063958 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:27.063963 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:27.063965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:27.064051 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:27.064375 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:27.064380 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:27.064382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:27.064613 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:43:27.064727 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 12:43:27.065189 (+ 462us) server_negotiation.cc:299] Negotiation successful
0927 12:43:27.065302 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 12:43:27.066221 14619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:27.053984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36945 (local address 127.0.98.3:50689)
0927 12:43:27.054645 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:27.054684 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 12:43:27.055702 (+ 1018us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:43:27.056382 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:43:27.056389 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:43:27.056745 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:43:27.057220 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:27.057233 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:27.058068 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:27.058073 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:27.058953 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:27.058962 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:27.059447 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:27.059451 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:27.059633 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:27.060182 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:43:27.060197 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:43:27.061927 (+ 1730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:43:27.063601 (+ 1674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:43:27.063606 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:43:27.063619 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:43:27.063856 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:43:27.064150 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:43:27.064152 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:43:27.064154 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:43:27.064282 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:43:27.064735 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:43:27.064742 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:43:27.064965 (+ 223us) client_negotiation.cc:769] Sending connection context
0927 12:43:27.065148 (+ 183us) client_negotiation.cc:240] Negotiation successful
0927 12:43:27.065440 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":119,"threads_started":1}
I20260927 12:43:27.067425 14617 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36945
I20260927 12:43:27.067679 14617 heartbeater.cc:464] Registering TS with master...
I20260927 12:43:27.068132 14617 heartbeater.cc:511] Master 127.0.98.62:36945 requested a full tablet report, sending...
I20260927 12:43:27.069130 14158 ts_manager.cc:194] Registered new tserver with Master: 02d683b804bd404aa1ea624a04611726 (127.0.98.3:35185)
I20260927 12:43:27.069754 14158 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50689
I20260927 12:43:27.080612 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 12:43:27.089561 14197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:27.082615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45322 (local address 127.0.98.62:36945)
0927 12:43:27.082727 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 12:43:27.082732 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:43:27.082897 (+ 165us) server_negotiation.cc:407] Connection header received
0927 12:43:27.083001 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:43:27.083003 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:43:27.083062 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:43:27.083136 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:43:27.083805 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:27.084311 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:27.085127 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:27.085329 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:27.085401 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:27.086741 (+ 1340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:43:27.086745 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:43:27.086747 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:43:27.086768 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:43:27.088175 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:27.088553 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:27.088555 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:27.088556 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:27.088622 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:27.088840 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:27.088842 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:27.088843 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:27.089033 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:43:27.089157 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0927 12:43:27.089359 (+ 202us) server_negotiation.cc:299] Negotiation successful
0927 12:43:27.089448 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260927 12:43:27.092393 14158 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45322:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 12:43:27.095376 14158 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:43:27.109467 14630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:27.103740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60028 (local address 127.0.98.1:38451)
0927 12:43:27.105111 (+ 1371us) server_negotiation.cc:206] Beginning negotiation
0927 12:43:27.105126 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 12:43:27.105148 (+ 22us) server_negotiation.cc:407] Connection header received
0927 12:43:27.105221 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:43:27.105226 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:43:27.105374 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:43:27.105541 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:43:27.105775 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:27.106510 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:27.108094 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:27.108801 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:27.108928 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:27.109095 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 12:43:27.109215 (+ 120us) server_negotiation.cc:299] Negotiation successful
0927 12:43:27.109309 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1264,"thread_start_us":118,"threads_started":1}
I20260927 12:43:27.109457 14629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:27.103459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38451 (local address 127.0.0.1:60028)
0927 12:43:27.103880 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:27.103924 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 12:43:27.104057 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:43:27.105510 (+ 1453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:43:27.105513 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:43:27.105531 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:43:27.105657 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:27.105664 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:27.106651 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:27.106657 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:27.107941 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:27.107952 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:27.108926 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:27.108929 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:27.109030 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:27.109067 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 12:43:27.109185 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 12:43:27.109309 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":87,"threads_started":1}
I20260927 12:43:27.110045 14631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:27.104169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35185 (local address 127.0.0.1:53430)
0927 12:43:27.104593 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:27.104607 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:43:27.104693 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:43:27.105330 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:43:27.105336 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:43:27.105370 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:43:27.105639 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:27.105647 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:27.107072 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:27.107076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:27.108311 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:27.108319 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:27.109589 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:27.109593 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:27.109714 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:27.109736 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 12:43:27.109805 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 12:43:27.109875 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"mutex_wait_us":138,"spinlock_wait_cycles":5760,"thread_start_us":117,"threads_started":1}
I20260927 12:43:27.110308 14632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:27.104132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43670 (local address 127.0.98.2:33847)
0927 12:43:27.104450 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0927 12:43:27.104463 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 12:43:27.104682 (+ 219us) server_negotiation.cc:407] Connection header received
0927 12:43:27.104767 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:43:27.104772 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:43:27.104981 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:43:27.105138 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:43:27.105775 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:27.106953 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:27.108475 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:27.109262 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:27.109371 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:27.109499 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 12:43:27.110076 (+ 577us) server_negotiation.cc:299] Negotiation successful
0927 12:43:27.110174 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":151,"threads_started":1}
I20260927 12:43:27.110636 14633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:27.104375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53430 (local address 127.0.98.3:35185)
0927 12:43:27.104808 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0927 12:43:27.104819 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 12:43:27.104836 (+ 17us) server_negotiation.cc:407] Connection header received
0927 12:43:27.104895 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:43:27.104899 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:43:27.105038 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:43:27.105169 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:43:27.105775 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:27.106929 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:27.108430 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:27.109263 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:27.109371 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:27.109499 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 12:43:27.110326 (+ 827us) server_negotiation.cc:299] Negotiation successful
0927 12:43:27.110460 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":141,"threads_started":1}
I20260927 12:43:27.110229 14628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:27.103513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33847 (local address 127.0.0.1:43670)
0927 12:43:27.104463 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:27.104478 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:43:27.104583 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:43:27.105330 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:43:27.105336 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:43:27.105372 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:43:27.105641 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:27.105647 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:27.107091 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:27.107094 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:27.108362 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:27.108372 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:27.109599 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:27.109603 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:27.109930 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:27.109950 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 12:43:27.110024 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 12:43:27.110095 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":893,"mutex_wait_us":384,"spinlock_wait_cycles":7424,"thread_start_us":120,"threads_started":1}
I20260927 12:43:27.112911 14283 tablet_service.cc:1467] Processing CreateTablet for tablet 2b048434037c4423bad26da0dde0a2e9 (DEFAULT_TABLE table=test-table [id=69dbd41351084281a326fdc185bf5e8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:43:27.113287 14551 tablet_service.cc:1467] Processing CreateTablet for tablet 2b048434037c4423bad26da0dde0a2e9 (DEFAULT_TABLE table=test-table [id=69dbd41351084281a326fdc185bf5e8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:43:27.114099 14283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b048434037c4423bad26da0dde0a2e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:43:27.113839 14417 tablet_service.cc:1467] Processing CreateTablet for tablet 2b048434037c4423bad26da0dde0a2e9 (DEFAULT_TABLE table=test-table [id=69dbd41351084281a326fdc185bf5e8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:43:27.114326 14551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b048434037c4423bad26da0dde0a2e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:43:27.114758 14417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b048434037c4423bad26da0dde0a2e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:43:27.120956 14634 tablet_bootstrap.cc:492] T 2b048434037c4423bad26da0dde0a2e9 P 02d683b804bd404aa1ea624a04611726: Bootstrap starting.
I20260927 12:43:27.121367 14635 tablet_bootstrap.cc:492] T 2b048434037c4423bad26da0dde0a2e9 P 3dc0e90d397a4c1bb3749bf9a7c50732: Bootstrap starting.
I20260927 12:43:27.122673 14636 tablet_bootstrap.cc:492] T 2b048434037c4423bad26da0dde0a2e9 P 79e57a29f88f433aa4f1215486f9545d: Bootstrap starting.
I20260927 12:43:27.123647 14635 tablet_bootstrap.cc:654] T 2b048434037c4423bad26da0dde0a2e9 P 3dc0e90d397a4c1bb3749bf9a7c50732: Neither blocks nor log segments found. Creating new log.
I20260927 12:43:27.123647 14634 tablet_bootstrap.cc:654] T 2b048434037c4423bad26da0dde0a2e9 P 02d683b804bd404aa1ea624a04611726: Neither blocks nor log segments found. Creating new log.
I20260927 12:43:27.124730 14636 tablet_bootstrap.cc:654] T 2b048434037c4423bad26da0dde0a2e9 P 79e57a29f88f433aa4f1215486f9545d: Neither blocks nor log segments found. Creating new log.
I20260927 12:43:27.125039 14635 log.cc:824] T 2b048434037c4423bad26da0dde0a2e9 P 3dc0e90d397a4c1bb3749bf9a7c50732: Log is configured to *not* fsync() on all Append() calls
I20260927 12:43:27.125028 14634 log.cc:824] T 2b048434037c4423bad26da0dde0a2e9 P 02d683b804bd404aa1ea624a04611726: Log is configured to *not* fsync() on all Append() calls
I20260927 12:43:27.125488 14636 log.cc:824] T 2b048434037c4423bad26da0dde0a2e9 P 79e57a29f88f433aa4f1215486f9545d: Log is configured to *not* fsync() on all Append() calls
I20260927 12:43:27.127096 14634 tablet_bootstrap.cc:492] T 2b048434037c4423bad26da0dde0a2e9 P 02d683b804bd404aa1ea624a04611726: No bootstrap required, opened a new log
I20260927 12:43:27.127096 14635 tablet_bootstrap.cc:492] T 2b048434037c4423bad26da0dde0a2e9 P 3dc0e90d397a4c1bb3749bf9a7c50732: No bootstrap required, opened a new log
I20260927 12:43:27.127274 14636 tablet_bootstrap.cc:492] T 2b048434037c4423bad26da0dde0a2e9 P 79e57a29f88f433aa4f1215486f9545d: No bootstrap required, opened a new log
I20260927 12:43:27.127290 14634 ts_tablet_manager.cc:1397] T 2b048434037c4423bad26da0dde0a2e9 P 02d683b804bd404aa1ea624a04611726: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260927 12:43:27.127290 14635 ts_tablet_manager.cc:1397] T 2b048434037c4423bad26da0dde0a2e9 P 3dc0e90d397a4c1bb3749bf9a7c50732: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 12:43:27.127478 14636 ts_tablet_manager.cc:1397] T 2b048434037c4423bad26da0dde0a2e9 P 79e57a29f88f433aa4f1215486f9545d: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 12:43:27.130400 14636 raft_consensus.cc:348] T 2b048434037c4423bad26da0dde0a2e9 P 79e57a29f88f433aa4f1215486f9545d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dc0e90d397a4c1bb3749bf9a7c50732" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38451 } } peers { permanent_uuid: "79e57a29f88f433aa4f1215486f9545d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33847 } } peers { permanent_uuid: "02d683b804bd404aa1ea624a04611726" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35185 } }
I20260927 12:43:27.130662 14636 raft_consensus.cc:374] T 2b048434037c4423bad26da0dde0a2e9 P 79e57a29f88f433aa4f1215486f9545d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:43:27.130713 14636 raft_consensus.cc:729] T 2b048434037c4423bad26da0dde0a2e9 P 79e57a29f88f433aa4f1215486f9545d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79e57a29f88f433aa4f1215486f9545d, State: Initialized, Role: FOLLOWER
I20260927 12:43:27.131201 14636 consensus_queue.cc:260] T 2b048434037c4423bad26da0dde0a2e9 P 79e57a29f88f433aa4f1215486f9545d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dc0e90d397a4c1bb3749bf9a7c50732" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38451 } } peers { permanent_uuid: "79e57a29f88f433aa4f1215486f9545d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33847 } } peers { permanent_uuid: "02d683b804bd404aa1ea624a04611726" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35185 } }
I20260927 12:43:27.131191 14635 raft_consensus.cc:348] T 2b048434037c4423bad26da0dde0a2e9 P 3dc0e90d397a4c1bb3749bf9a7c50732 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dc0e90d397a4c1bb3749bf9a7c50732" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38451 } } peers { permanent_uuid: "79e57a29f88f433aa4f1215486f9545d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33847 } } peers { permanent_uuid: "02d683b804bd404aa1ea624a04611726" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35185 } }
I20260927 12:43:27.131194 14634 raft_consensus.cc:348] T 2b048434037c4423bad26da0dde0a2e9 P 02d683b804bd404aa1ea624a04611726 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dc0e90d397a4c1bb3749bf9a7c50732" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38451 } } peers { permanent_uuid: "79e57a29f88f433aa4f1215486f9545d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33847 } } peers { permanent_uuid: "02d683b804bd404aa1ea624a04611726" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35185 } }
I20260927 12:43:27.131428 14635 raft_consensus.cc:374] T 2b048434037c4423bad26da0dde0a2e9 P 3dc0e90d397a4c1bb3749bf9a7c50732 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:43:27.131428 14634 raft_consensus.cc:374] T 2b048434037c4423bad26da0dde0a2e9 P 02d683b804bd404aa1ea624a04611726 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:43:27.131479 14635 raft_consensus.cc:729] T 2b048434037c4423bad26da0dde0a2e9 P 3dc0e90d397a4c1bb3749bf9a7c50732 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dc0e90d397a4c1bb3749bf9a7c50732, State: Initialized, Role: FOLLOWER
I20260927 12:43:27.131488 14634 raft_consensus.cc:729] T 2b048434037c4423bad26da0dde0a2e9 P 02d683b804bd404aa1ea624a04611726 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02d683b804bd404aa1ea624a04611726, State: Initialized, Role: FOLLOWER
I20260927 12:43:27.131932 14634 consensus_queue.cc:260] T 2b048434037c4423bad26da0dde0a2e9 P 02d683b804bd404aa1ea624a04611726 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dc0e90d397a4c1bb3749bf9a7c50732" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38451 } } peers { permanent_uuid: "79e57a29f88f433aa4f1215486f9545d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33847 } } peers { permanent_uuid: "02d683b804bd404aa1ea624a04611726" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35185 } }
I20260927 12:43:27.131932 14635 consensus_queue.cc:260] T 2b048434037c4423bad26da0dde0a2e9 P 3dc0e90d397a4c1bb3749bf9a7c50732 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dc0e90d397a4c1bb3749bf9a7c50732" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38451 } } peers { permanent_uuid: "79e57a29f88f433aa4f1215486f9545d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33847 } } peers { permanent_uuid: "02d683b804bd404aa1ea624a04611726" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35185 } }
I20260927 12:43:27.132140 14636 ts_tablet_manager.cc:1428] T 2b048434037c4423bad26da0dde0a2e9 P 79e57a29f88f433aa4f1215486f9545d: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 12:43:27.132535 14483 heartbeater.cc:503] Master 127.0.98.62:36945 was elected leader, sending a full tablet report...
I20260927 12:43:27.132880 14634 ts_tablet_manager.cc:1428] T 2b048434037c4423bad26da0dde0a2e9 P 02d683b804bd404aa1ea624a04611726: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 12:43:27.133188 14617 heartbeater.cc:503] Master 127.0.98.62:36945 was elected leader, sending a full tablet report...
I20260927 12:43:27.133927 14635 ts_tablet_manager.cc:1428] T 2b048434037c4423bad26da0dde0a2e9 P 3dc0e90d397a4c1bb3749bf9a7c50732: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260927 12:43:27.134649 14349 heartbeater.cc:503] Master 127.0.98.62:36945 was elected leader, sending a full tablet report...
W20260927 12:43:27.302728 14618 tablet.cc:2369] T 2b048434037c4423bad26da0dde0a2e9 P 02d683b804bd404aa1ea624a04611726: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 12:43:27.348685 14484 tablet.cc:2369] T 2b048434037c4423bad26da0dde0a2e9 P 79e57a29f88f433aa4f1215486f9545d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 12:43:27.380587 14350 tablet.cc:2369] T 2b048434037c4423bad26da0dde0a2e9 P 3dc0e90d397a4c1bb3749bf9a7c50732: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:43:27.447993 14642 raft_consensus.cc:482] T 2b048434037c4423bad26da0dde0a2e9 P 02d683b804bd404aa1ea624a04611726 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:43:27.448197 14642 raft_consensus.cc:504] T 2b048434037c4423bad26da0dde0a2e9 P 02d683b804bd404aa1ea624a04611726 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dc0e90d397a4c1bb3749bf9a7c50732" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38451 } } peers { permanent_uuid: "79e57a29f88f433aa4f1215486f9545d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33847 } } peers { permanent_uuid: "02d683b804bd404aa1ea624a04611726" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35185 } }
I20260927 12:43:27.449314 14642 leader_election.cc:288] T 2b048434037c4423bad26da0dde0a2e9 P 02d683b804bd404aa1ea624a04611726 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3dc0e90d397a4c1bb3749bf9a7c50732 (127.0.98.1:38451), 79e57a29f88f433aa4f1215486f9545d (127.0.98.2:33847)
I20260927 12:43:27.454237 14630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:27.449737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59709 (local address 127.0.98.1:38451)
0927 12:43:27.449876 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0927 12:43:27.449883 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 12:43:27.449905 (+ 22us) server_negotiation.cc:407] Connection header received
0927 12:43:27.449996 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:43:27.450000 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:43:27.450075 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:43:27.450147 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:43:27.450510 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:27.451269 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:27.452842 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:27.453443 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:27.453585 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:27.453646 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 12:43:27.454033 (+ 387us) server_negotiation.cc:299] Negotiation successful
0927 12:43:27.454109 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 12:43:27.454237 14632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:27.449741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35867 (local address 127.0.98.2:33847)
0927 12:43:27.449883 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0927 12:43:27.449888 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:43:27.450280 (+ 392us) server_negotiation.cc:407] Connection header received
0927 12:43:27.450385 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:43:27.450387 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:43:27.450427 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:43:27.450525 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:43:27.450808 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:27.451490 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:27.452842 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:27.453443 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:27.453558 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:27.453646 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 12:43:27.454009 (+ 363us) server_negotiation.cc:299] Negotiation successful
0927 12:43:27.454111 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 12:43:27.454244 14619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:27.449626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38451 (local address 127.0.98.3:59709)
0927 12:43:27.449753 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:27.449771 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 12:43:27.449886 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:43:27.450196 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:43:27.450201 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:43:27.450224 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:43:27.450385 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:27.450399 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:27.451405 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:27.451408 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:27.452706 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:27.452716 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:27.453622 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:27.453626 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:27.453846 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:27.453895 (+ 49us) client_negotiation.cc:769] Sending connection context
0927 12:43:27.454012 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 12:43:27.454094 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":111,"spinlock_wait_cycles":24448}
I20260927 12:43:27.454253 14643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:27.449626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33847 (local address 127.0.98.3:35867)
0927 12:43:27.450174 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:27.450186 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 12:43:27.450291 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:43:27.450583 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:43:27.450586 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:43:27.450607 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:43:27.450701 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:27.450705 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:27.451608 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:27.451611 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:27.452724 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:27.452731 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:27.453622 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:27.453626 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:27.453819 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:27.453888 (+ 69us) client_negotiation.cc:769] Sending connection context
0927 12:43:27.453977 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 12:43:27.454079 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"mutex_wait_us":220,"spinlock_wait_cycles":2688,"thread_start_us":112,"threads_started":1}
I20260927 12:43:27.454962 14437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b048434037c4423bad26da0dde0a2e9" candidate_uuid: "02d683b804bd404aa1ea624a04611726" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79e57a29f88f433aa4f1215486f9545d" is_pre_election: true
I20260927 12:43:27.454979 14303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b048434037c4423bad26da0dde0a2e9" candidate_uuid: "02d683b804bd404aa1ea624a04611726" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3dc0e90d397a4c1bb3749bf9a7c50732" is_pre_election: true
I20260927 12:43:27.455272 14437 raft_consensus.cc:2454] T 2b048434037c4423bad26da0dde0a2e9 P 79e57a29f88f433aa4f1215486f9545d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02d683b804bd404aa1ea624a04611726 in term 0.
I20260927 12:43:27.455286 14303 raft_consensus.cc:2454] T 2b048434037c4423bad26da0dde0a2e9 P 3dc0e90d397a4c1bb3749bf9a7c50732 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02d683b804bd404aa1ea624a04611726 in term 0.
I20260927 12:43:27.455747 14507 leader_election.cc:302] T 2b048434037c4423bad26da0dde0a2e9 P 02d683b804bd404aa1ea624a04611726 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02d683b804bd404aa1ea624a04611726, 79e57a29f88f433aa4f1215486f9545d; no voters:
I20260927 12:43:27.456044 14642 raft_consensus.cc:2781] T 2b048434037c4423bad26da0dde0a2e9 P 02d683b804bd404aa1ea624a04611726 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:43:27.456133 14642 raft_consensus.cc:482] T 2b048434037c4423bad26da0dde0a2e9 P 02d683b804bd404aa1ea624a04611726 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:43:27.456175 14642 raft_consensus.cc:3037] T 2b048434037c4423bad26da0dde0a2e9 P 02d683b804bd404aa1ea624a04611726 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:43:27.457192 14642 raft_consensus.cc:504] T 2b048434037c4423bad26da0dde0a2e9 P 02d683b804bd404aa1ea624a04611726 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dc0e90d397a4c1bb3749bf9a7c50732" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38451 } } peers { permanent_uuid: "79e57a29f88f433aa4f1215486f9545d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33847 } } peers { permanent_uuid: "02d683b804bd404aa1ea624a04611726" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35185 } }
I20260927 12:43:27.457521 14642 leader_election.cc:288] T 2b048434037c4423bad26da0dde0a2e9 P 02d683b804bd404aa1ea624a04611726 [CANDIDATE]: Term 1 election: Requested vote from peers 3dc0e90d397a4c1bb3749bf9a7c50732 (127.0.98.1:38451), 79e57a29f88f433aa4f1215486f9545d (127.0.98.2:33847)
I20260927 12:43:27.457866 14303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b048434037c4423bad26da0dde0a2e9" candidate_uuid: "02d683b804bd404aa1ea624a04611726" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3dc0e90d397a4c1bb3749bf9a7c50732"
I20260927 12:43:27.457867 14437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b048434037c4423bad26da0dde0a2e9" candidate_uuid: "02d683b804bd404aa1ea624a04611726" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79e57a29f88f433aa4f1215486f9545d"
I20260927 12:43:27.458029 14437 raft_consensus.cc:3037] T 2b048434037c4423bad26da0dde0a2e9 P 79e57a29f88f433aa4f1215486f9545d [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:43:27.457970 14303 raft_consensus.cc:3037] T 2b048434037c4423bad26da0dde0a2e9 P 3dc0e90d397a4c1bb3749bf9a7c50732 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:43:27.459007 14303 raft_consensus.cc:2454] T 2b048434037c4423bad26da0dde0a2e9 P 3dc0e90d397a4c1bb3749bf9a7c50732 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02d683b804bd404aa1ea624a04611726 in term 1.
I20260927 12:43:27.459087 14437 raft_consensus.cc:2454] T 2b048434037c4423bad26da0dde0a2e9 P 79e57a29f88f433aa4f1215486f9545d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02d683b804bd404aa1ea624a04611726 in term 1.
I20260927 12:43:27.459336 14504 leader_election.cc:302] T 2b048434037c4423bad26da0dde0a2e9 P 02d683b804bd404aa1ea624a04611726 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02d683b804bd404aa1ea624a04611726, 3dc0e90d397a4c1bb3749bf9a7c50732; no voters:
I20260927 12:43:27.459527 14642 raft_consensus.cc:2781] T 2b048434037c4423bad26da0dde0a2e9 P 02d683b804bd404aa1ea624a04611726 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:43:27.459782 14642 raft_consensus.cc:686] T 2b048434037c4423bad26da0dde0a2e9 P 02d683b804bd404aa1ea624a04611726 [term 1 LEADER]: Becoming Leader. State: Replica: 02d683b804bd404aa1ea624a04611726, State: Running, Role: LEADER
I20260927 12:43:27.460131 14642 consensus_queue.cc:237] T 2b048434037c4423bad26da0dde0a2e9 P 02d683b804bd404aa1ea624a04611726 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dc0e90d397a4c1bb3749bf9a7c50732" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38451 } } peers { permanent_uuid: "79e57a29f88f433aa4f1215486f9545d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33847 } } peers { permanent_uuid: "02d683b804bd404aa1ea624a04611726" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35185 } }
I20260927 12:43:27.463560 14157 catalog_manager.cc:5495] T 2b048434037c4423bad26da0dde0a2e9 P 02d683b804bd404aa1ea624a04611726 reported cstate change: term changed from 0 to 1, leader changed from <none> to 02d683b804bd404aa1ea624a04611726 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "02d683b804bd404aa1ea624a04611726" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dc0e90d397a4c1bb3749bf9a7c50732" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79e57a29f88f433aa4f1215486f9545d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02d683b804bd404aa1ea624a04611726" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35185 } health_report { overall_health: HEALTHY } } }
I20260927 12:43:27.561122 14633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:27.557553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53438 (local address 127.0.98.3:35185)
0927 12:43:27.557734 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 12:43:27.557737 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:43:27.557751 (+ 14us) server_negotiation.cc:407] Connection header received
0927 12:43:27.557872 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:43:27.557874 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:43:27.557930 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:43:27.558021 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:43:27.558256 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:27.558750 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:27.559547 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:27.559807 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:27.559886 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:27.560172 (+ 286us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:43:27.560641 (+ 469us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:43:27.560761 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 12:43:27.560913 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 12:43:27.560972 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 12:43:27.567005 14437 raft_consensus.cc:1261] T 2b048434037c4423bad26da0dde0a2e9 P 79e57a29f88f433aa4f1215486f9545d [term 1 FOLLOWER]: Refusing update from remote peer 02d683b804bd404aa1ea624a04611726: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:43:27.567003 14303 raft_consensus.cc:1261] T 2b048434037c4423bad26da0dde0a2e9 P 3dc0e90d397a4c1bb3749bf9a7c50732 [term 1 FOLLOWER]: Refusing update from remote peer 02d683b804bd404aa1ea624a04611726: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:43:27.567761 14644 consensus_queue.cc:1035] T 2b048434037c4423bad26da0dde0a2e9 P 02d683b804bd404aa1ea624a04611726 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3dc0e90d397a4c1bb3749bf9a7c50732" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:43:27.567947 14642 consensus_queue.cc:1035] T 2b048434037c4423bad26da0dde0a2e9 P 02d683b804bd404aa1ea624a04611726 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e57a29f88f433aa4f1215486f9545d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:43:27.575438 14648 mvcc.cc:205] Tried to move back new op lower bound from 7333941278986801152 to 7333941278559260672. Current Snapshot: MvccSnapshot[applied={T|T < 7333941278986801152}]
I20260927 12:43:27.575932 14647 mvcc.cc:205] Tried to move back new op lower bound from 7333941278986801152 to 7333941278559260672. Current Snapshot: MvccSnapshot[applied={T|T < 7333941278986801152}]
I20260927 12:43:27.578234 14649 mvcc.cc:205] Tried to move back new op lower bound from 7333941278986801152 to 7333941278559260672. Current Snapshot: MvccSnapshot[applied={T|T < 7333941278986801152}]
I20260927 12:43:27.583596 14156 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:43:27.583765 14156 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:43:27.586131 14156 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4027986ce39f4ac898f9f1f7b7a98a93: Sending DeleteTablet for 3 replicas of tablet 2b048434037c4423bad26da0dde0a2e9
I20260927 12:43:27.586977 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 14220
I20260927 12:43:27.586963 14417 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b048434037c4423bad26da0dde0a2e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:43:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43670
I20260927 12:43:27.586963 14283 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b048434037c4423bad26da0dde0a2e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:43:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60028
I20260927 12:43:27.587590 14551 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b048434037c4423bad26da0dde0a2e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:43:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53430
I20260927 12:43:27.587954 14658 tablet_replica.cc:330] T 2b048434037c4423bad26da0dde0a2e9 P 02d683b804bd404aa1ea624a04611726: stopping tablet replica
I20260927 12:43:27.588249 14658 raft_consensus.cc:2229] T 2b048434037c4423bad26da0dde0a2e9 P 02d683b804bd404aa1ea624a04611726 [term 1 LEADER]: Raft consensus shutting down.
I20260927 12:43:27.588587 14658 raft_consensus.cc:2258] T 2b048434037c4423bad26da0dde0a2e9 P 02d683b804bd404aa1ea624a04611726 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:43:27.589686 14658 ts_tablet_manager.cc:1905] T 2b048434037c4423bad26da0dde0a2e9 P 02d683b804bd404aa1ea624a04611726: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:43:27.589772 14657 tablet_replica.cc:330] T 2b048434037c4423bad26da0dde0a2e9 P 79e57a29f88f433aa4f1215486f9545d: stopping tablet replica
I20260927 12:43:27.590026 14657 raft_consensus.cc:2229] T 2b048434037c4423bad26da0dde0a2e9 P 79e57a29f88f433aa4f1215486f9545d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:43:27.590330 14657 raft_consensus.cc:2258] T 2b048434037c4423bad26da0dde0a2e9 P 79e57a29f88f433aa4f1215486f9545d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:43:27.592191 14658 ts_tablet_manager.cc:1918] T 2b048434037c4423bad26da0dde0a2e9 P 02d683b804bd404aa1ea624a04611726: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:43:27.592260 14658 log.cc:1196] T 2b048434037c4423bad26da0dde0a2e9 P 02d683b804bd404aa1ea624a04611726: Deleting WAL directory at /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-2/wal/wals/2b048434037c4423bad26da0dde0a2e9
I20260927 12:43:27.592475 14658 ts_tablet_manager.cc:1939] T 2b048434037c4423bad26da0dde0a2e9 P 02d683b804bd404aa1ea624a04611726: Deleting consensus metadata
I20260927 12:43:27.593773 14145 catalog_manager.cc:4841] TS 02d683b804bd404aa1ea624a04611726 (127.0.98.3:35185): tablet 2b048434037c4423bad26da0dde0a2e9 (table test-table [id=69dbd41351084281a326fdc185bf5e8d]) successfully deleted
I20260927 12:43:27.594997 14657 ts_tablet_manager.cc:1905] T 2b048434037c4423bad26da0dde0a2e9 P 79e57a29f88f433aa4f1215486f9545d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 12:43:27.595881 14142 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38451 (error 108)
I20260927 12:43:27.597016 14628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:27.596681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38451 (local address 127.0.0.1:60044)
0927 12:43:27.596863 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:27.596933 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260927 12:43:27.597285 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 14354
W20260927 12:43:27.597266 14142 catalog_manager.cc:4584] TS 3dc0e90d397a4c1bb3749bf9a7c50732 (127.0.98.1:38451): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2b048434037c4423bad26da0dde0a2e9: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38451: connect: Connection refused (error 111)
I20260927 12:43:27.603149 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 14488
I20260927 12:43:27.605113 14628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:27.604826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33847 (local address 127.0.0.1:43682)
0927 12:43:27.604958 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:27.605010 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:33847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 12:43:27.608354 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 14126
I20260927 12:43:27.614459 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36945
--webserver_interface=127.0.98.62
--webserver_port=45379
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36945
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:43:27.725006 14660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:43:27.725291 14660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:43:27.725332 14660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:43:27.729475 14660 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:43:27.729548 14660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:43:27.729592 14660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:43:27.729610 14660 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:43:27.729625 14660 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:43:27.735097 14660 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36945
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=45379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14660
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:43:27.736202 14660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:43:27.737255 14660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:43:27.749977 14666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:43:27.749977 14667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:43:27.750065 14669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:43:27.750300 14660 server_base.cc:1034] running on GCE node
I20260927 12:43:27.750921 14660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:43:27.752213 14660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:43:27.753412 14660 hybrid_clock.cc:648] HybridClock initialized: now 1790513007753373 us; error 50 us; skew 500 ppm
Sep 27 12:43:27 dist-test-slave-bnc1 krb5kdc[14113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513007, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:43:27.756541 14660 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:43:27.757642 14660 webserver.cc:460] Webserver started at http://127.0.98.62:45379/ using document root <none> and password file <none>
I20260927 12:43:27.758276 14660 fs_manager.cc:359] Metadata directory not provided
I20260927 12:43:27.758392 14660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:43:27.762549 14660 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 12:43:27.764760 14676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:43:27.766166 14660 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:43:27.766312 14660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/master-0/wal
uuid: "4027986ce39f4ac898f9f1f7b7a98a93"
format_stamp: "Formatted at 2026-09-27 12:43:26 on dist-test-slave-bnc1"
server_key: "8d9e049b3635deaa906bc249e97dca15"
server_key_iv: "f6360a5d8bbd11b7e31d6a6f64f31fd6"
server_key_version: "encryptionkey@0"
I20260927 12:43:27.766920 14660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:43:27.786007 14660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:43:27.789654 14660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:43:27.789866 14660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:43:27.797199 14660 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36945
I20260927 12:43:27.797478 14728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36945 every 8 connection(s)
I20260927 12:43:27.798338 14660 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:43:27.801048 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 14660
I20260927 12:43:27.801519 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:38451
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:36945
--webserver_port=43873
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 12:43:27.803236 14729 sys_catalog.cc:263] Verifying existing consensus state
I20260927 12:43:27.804910 14729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4027986ce39f4ac898f9f1f7b7a98a93: Bootstrap starting.
I20260927 12:43:27.821970 14729 log.cc:824] T 00000000000000000000000000000000 P 4027986ce39f4ac898f9f1f7b7a98a93: Log is configured to *not* fsync() on all Append() calls
I20260927 12:43:27.830521 14729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4027986ce39f4ac898f9f1f7b7a98a93: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 12:43:27.830945 14729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4027986ce39f4ac898f9f1f7b7a98a93: Bootstrap complete.
I20260927 12:43:27.835072 14729 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4027986ce39f4ac898f9f1f7b7a98a93 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4027986ce39f4ac898f9f1f7b7a98a93" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36945 } }
I20260927 12:43:27.835471 14729 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4027986ce39f4ac898f9f1f7b7a98a93 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4027986ce39f4ac898f9f1f7b7a98a93, State: Initialized, Role: FOLLOWER
I20260927 12:43:27.836025 14729 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4027986ce39f4ac898f9f1f7b7a98a93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4027986ce39f4ac898f9f1f7b7a98a93" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36945 } }
I20260927 12:43:27.836148 14729 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4027986ce39f4ac898f9f1f7b7a98a93 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:43:27.836236 14729 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4027986ce39f4ac898f9f1f7b7a98a93 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:43:27.836324 14729 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4027986ce39f4ac898f9f1f7b7a98a93 [term 1 FOLLOWER]: Advancing to term 2
I20260927 12:43:27.838249 14729 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4027986ce39f4ac898f9f1f7b7a98a93 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4027986ce39f4ac898f9f1f7b7a98a93" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36945 } }
I20260927 12:43:27.838627 14729 leader_election.cc:302] T 00000000000000000000000000000000 P 4027986ce39f4ac898f9f1f7b7a98a93 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4027986ce39f4ac898f9f1f7b7a98a93; no voters:
I20260927 12:43:27.838951 14729 leader_election.cc:288] T 00000000000000000000000000000000 P 4027986ce39f4ac898f9f1f7b7a98a93 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 12:43:27.839104 14733 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4027986ce39f4ac898f9f1f7b7a98a93 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 12:43:27.839354 14733 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4027986ce39f4ac898f9f1f7b7a98a93 [term 2 LEADER]: Becoming Leader. State: Replica: 4027986ce39f4ac898f9f1f7b7a98a93, State: Running, Role: LEADER
I20260927 12:43:27.839671 14733 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4027986ce39f4ac898f9f1f7b7a98a93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4027986ce39f4ac898f9f1f7b7a98a93" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36945 } }
I20260927 12:43:27.840204 14729 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4027986ce39f4ac898f9f1f7b7a98a93 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:43:27.841220 14735 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4027986ce39f4ac898f9f1f7b7a98a93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4027986ce39f4ac898f9f1f7b7a98a93. Latest consensus state: current_term: 2 leader_uuid: "4027986ce39f4ac898f9f1f7b7a98a93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4027986ce39f4ac898f9f1f7b7a98a93" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36945 } } }
I20260927 12:43:27.841361 14735 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4027986ce39f4ac898f9f1f7b7a98a93 [sys.catalog]: This master's current role is: LEADER
I20260927 12:43:27.841634 14734 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4027986ce39f4ac898f9f1f7b7a98a93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4027986ce39f4ac898f9f1f7b7a98a93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4027986ce39f4ac898f9f1f7b7a98a93" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36945 } } }
I20260927 12:43:27.841724 14734 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4027986ce39f4ac898f9f1f7b7a98a93 [sys.catalog]: This master's current role is: LEADER
I20260927 12:43:27.841815 14742 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:43:27.847081 14742 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:43:27.847781 14742 catalog_manager.cc:1259] Loaded cluster ID: 102886e1aa324ab1bf5890f54a30055d
I20260927 12:43:27.847842 14742 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:43:27.849160 14742 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260927 12:43:27.853610 14742 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260927 12:43:27.859072 14742 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:43:27.859800 14742 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4027986ce39f4ac898f9f1f7b7a98a93: Loaded TSK: 0
I20260927 12:43:27.861043 14742 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 12:43:27.921022 14731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:43:27.921288 14731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:43:27.921331 14731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:43:27.925380 14731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:43:27.925448 14731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:43:27.925508 14731 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:43:27.930913 14731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:38451
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=43873
--tserver_master_addrs=127.0.98.62:36945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:43:27.931905 14731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:43:27.932890 14731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:43:27.943323 14756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:43:27.943364 14757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:43:27.943574 14759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:43:27.943723 14731 server_base.cc:1034] running on GCE node
I20260927 12:43:27.944285 14731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:43:27.945124 14731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:43:27.948434 14731 hybrid_clock.cc:648] HybridClock initialized: now 1790513007948391 us; error 49 us; skew 500 ppm
Sep 27 12:43:27 dist-test-slave-bnc1 krb5kdc[14113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513007, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:43:27.951301 14731 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 12:43:27.952201 14731 webserver.cc:460] Webserver started at http://127.0.98.1:43873/ using document root <none> and password file <none>
I20260927 12:43:27.952741 14731 fs_manager.cc:359] Metadata directory not provided
I20260927 12:43:27.952812 14731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:43:27.956647 14731 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:43:27.958921 14766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:43:27.960276 14731 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 12:43:27.960429 14731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-0/wal
uuid: "3dc0e90d397a4c1bb3749bf9a7c50732"
format_stamp: "Formatted at 2026-09-27 12:43:26 on dist-test-slave-bnc1"
server_key: "4b83bf9be58a116b974810c085611cce"
server_key_iv: "3532482f32615bc41ad994ad71f535f9"
server_key_version: "encryptionkey@0"
I20260927 12:43:27.961077 14731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:43:27.981638 14731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:43:27.984845 14731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:43:27.985075 14731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:43:27.985750 14731 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:43:27.986842 14773 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 12:43:27.990726 14731 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 12:43:27.990810 14731 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260927 12:43:27.990916 14731 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 12:43:27.992650 14731 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 12:43:27.992724 14731 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260927 12:43:27.992959 14773 tablet_bootstrap.cc:492] T 2b048434037c4423bad26da0dde0a2e9 P 3dc0e90d397a4c1bb3749bf9a7c50732: Bootstrap starting.
I20260927 12:43:28.003338 14731 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38451
I20260927 12:43:28.003378 14880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38451 every 8 connection(s)
I20260927 12:43:28.004304 14731 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:43:28.008447 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 14731
I20260927 12:43:28.008889 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:33847
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:36945
--webserver_port=35803
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 12:43:28.013204 14773 log.cc:824] T 2b048434037c4423bad26da0dde0a2e9 P 3dc0e90d397a4c1bb3749bf9a7c50732: Log is configured to *not* fsync() on all Append() calls
Sep 27 12:43:28 dist-test-slave-bnc1 krb5kdc[14113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513007, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:43:28.020661 14773 tablet_bootstrap.cc:492] T 2b048434037c4423bad26da0dde0a2e9 P 3dc0e90d397a4c1bb3749bf9a7c50732: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 12:43:28.021039 14773 tablet_bootstrap.cc:492] T 2b048434037c4423bad26da0dde0a2e9 P 3dc0e90d397a4c1bb3749bf9a7c50732: Bootstrap complete.
I20260927 12:43:28.022161 14773 ts_tablet_manager.cc:1397] T 2b048434037c4423bad26da0dde0a2e9 P 3dc0e90d397a4c1bb3749bf9a7c50732: Time spent bootstrapping tablet: real 0.029s user 0.021s sys 0.007s
I20260927 12:43:28.023502 14883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:28.006517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36945 (local address 127.0.98.1:36821)
0927 12:43:28.006989 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:28.007040 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 12:43:28.007846 (+ 806us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:43:28.009680 (+ 1834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:43:28.009689 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:43:28.010147 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:43:28.010887 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:28.010908 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:28.012299 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:28.012302 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:28.013236 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:28.013246 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:28.014223 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:28.014231 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:28.014471 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:28.015205 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:43:28.015233 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:43:28.017350 (+ 2117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:43:28.020194 (+ 2844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:43:28.020202 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:43:28.020218 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:43:28.020587 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:43:28.021003 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:43:28.021006 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:43:28.021008 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:43:28.021174 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:43:28.021847 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:43:28.021853 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:43:28.022164 (+ 311us) client_negotiation.cc:769] Sending connection context
0927 12:43:28.022328 (+ 164us) client_negotiation.cc:240] Negotiation successful
0927 12:43:28.022620 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":148,"threads_started":1}
I20260927 12:43:28.024533 14884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:28.006576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36821 (local address 127.0.98.62:36945)
0927 12:43:28.007096 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0927 12:43:28.007115 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 12:43:28.007162 (+ 47us) server_negotiation.cc:407] Connection header received
0927 12:43:28.008299 (+ 1137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:43:28.008314 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:43:28.008718 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:43:28.009100 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:43:28.011086 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:28.012126 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:28.013398 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:28.014064 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:28.014227 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:28.017617 (+ 3390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:43:28.017627 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:43:28.017644 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:43:28.017683 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:43:28.020000 (+ 2317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:28.020747 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:28.020752 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:28.020758 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:28.020895 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:28.021285 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:28.021288 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:28.021290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:28.021718 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:43:28.023026 (+ 1308us) server_negotiation.cc:1035] Waiting for connection context
0927 12:43:28.023181 (+ 155us) server_negotiation.cc:299] Negotiation successful
0927 12:43:28.023457 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":141,"threads_started":1}
I20260927 12:43:28.024828 14773 raft_consensus.cc:348] T 2b048434037c4423bad26da0dde0a2e9 P 3dc0e90d397a4c1bb3749bf9a7c50732 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dc0e90d397a4c1bb3749bf9a7c50732" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38451 } } peers { permanent_uuid: "79e57a29f88f433aa4f1215486f9545d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33847 } } peers { permanent_uuid: "02d683b804bd404aa1ea624a04611726" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35185 } }
I20260927 12:43:28.025301 14773 raft_consensus.cc:729] T 2b048434037c4423bad26da0dde0a2e9 P 3dc0e90d397a4c1bb3749bf9a7c50732 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dc0e90d397a4c1bb3749bf9a7c50732, State: Initialized, Role: FOLLOWER
I20260927 12:43:28.025776 14773 consensus_queue.cc:260] T 2b048434037c4423bad26da0dde0a2e9 P 3dc0e90d397a4c1bb3749bf9a7c50732 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dc0e90d397a4c1bb3749bf9a7c50732" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38451 } } peers { permanent_uuid: "79e57a29f88f433aa4f1215486f9545d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33847 } } peers { permanent_uuid: "02d683b804bd404aa1ea624a04611726" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35185 } }
I20260927 12:43:28.026711 14881 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36945
I20260927 12:43:28.026799 14773 ts_tablet_manager.cc:1428] T 2b048434037c4423bad26da0dde0a2e9 P 3dc0e90d397a4c1bb3749bf9a7c50732: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260927 12:43:28.026930 14881 heartbeater.cc:464] Registering TS with master...
I20260927 12:43:28.027474 14881 heartbeater.cc:511] Master 127.0.98.62:36945 requested a full tablet report, sending...
I20260927 12:43:28.030064 14693 ts_manager.cc:194] Registered new tserver with Master: 3dc0e90d397a4c1bb3749bf9a7c50732 (127.0.98.1:38451)
I20260927 12:43:28.033857 14693 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36821
I20260927 12:43:28.036375 14881 heartbeater.cc:503] Master 127.0.98.62:36945 was elected leader, sending a full tablet report...
Sep 27 12:43:28 dist-test-slave-bnc1 krb5kdc[14113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513007, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260927 12:43:28.044250 14888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:28.033208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38451 (local address 127.0.0.1:60046)
0927 12:43:28.033643 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:28.033686 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 12:43:28.033802 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:43:28.034853 (+ 1051us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:43:28.034862 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:43:28.035183 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:43:28.035274 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:28.035280 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:28.036216 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:28.036221 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:28.037001 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:28.037010 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:28.037727 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:28.037742 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:28.037851 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:28.038425 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:43:28.038437 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:43:28.039983 (+ 1546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:43:28.042514 (+ 2531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:43:28.042522 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:43:28.042525 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:43:28.042755 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:43:28.043046 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:43:28.043048 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:43:28.043050 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:43:28.043132 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:43:28.043616 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:43:28.043621 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:43:28.043739 (+ 118us) client_negotiation.cc:769] Sending connection context
0927 12:43:28.043880 (+ 141us) client_negotiation.cc:240] Negotiation successful
0927 12:43:28.044069 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":130,"threads_started":1}
I20260927 12:43:28.044262 14889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:28.033664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60046 (local address 127.0.98.1:38451)
0927 12:43:28.034233 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0927 12:43:28.034243 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 12:43:28.034265 (+ 22us) server_negotiation.cc:407] Connection header received
0927 12:43:28.034327 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:43:28.034332 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:43:28.034532 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:43:28.034696 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:43:28.035402 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:28.036100 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:28.037121 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:28.037625 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:28.037945 (+ 320us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:28.040105 (+ 2160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:43:28.040113 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:43:28.040125 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:43:28.040150 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:43:28.042413 (+ 2263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:28.042862 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:28.042868 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:28.042870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:28.042958 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:28.043240 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:28.043244 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:28.043245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:28.043484 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:43:28.043627 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 12:43:28.043958 (+ 331us) server_negotiation.cc:299] Negotiation successful
0927 12:43:28.044121 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":79,"threads_started":1}
I20260927 12:43:28.045151 14815 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b048434037c4423bad26da0dde0a2e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:43:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60046
I20260927 12:43:28.046005 14890 tablet_replica.cc:330] stopping tablet replica
I20260927 12:43:28.046336 14890 raft_consensus.cc:2229] T 2b048434037c4423bad26da0dde0a2e9 P 3dc0e90d397a4c1bb3749bf9a7c50732 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:43:28.046584 14890 raft_consensus.cc:2258] T 2b048434037c4423bad26da0dde0a2e9 P 3dc0e90d397a4c1bb3749bf9a7c50732 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:43:28.047734 14890 ts_tablet_manager.cc:1905] T 2b048434037c4423bad26da0dde0a2e9 P 3dc0e90d397a4c1bb3749bf9a7c50732: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:43:28.052152 14890 ts_tablet_manager.cc:1918] T 2b048434037c4423bad26da0dde0a2e9 P 3dc0e90d397a4c1bb3749bf9a7c50732: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:43:28.052255 14890 log.cc:1196] T 2b048434037c4423bad26da0dde0a2e9 P 3dc0e90d397a4c1bb3749bf9a7c50732: Deleting WAL directory at /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-0/wal/wals/2b048434037c4423bad26da0dde0a2e9
I20260927 12:43:28.052531 14890 ts_tablet_manager.cc:1939] T 2b048434037c4423bad26da0dde0a2e9 P 3dc0e90d397a4c1bb3749bf9a7c50732: Deleting consensus metadata
I20260927 12:43:28.054428 14677 catalog_manager.cc:4841] TS 3dc0e90d397a4c1bb3749bf9a7c50732 (127.0.98.1:38451): tablet 2b048434037c4423bad26da0dde0a2e9 (table test-table [id=69dbd41351084281a326fdc185bf5e8d]) successfully deleted
W20260927 12:43:28.128611 14885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:43:28.128867 14885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:43:28.128911 14885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:43:28.132952 14885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:43:28.133018 14885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:43:28.133075 14885 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:43:28.138391 14885 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:33847
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=35803
--tserver_master_addrs=127.0.98.62:36945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14885
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:43:28.139442 14885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:43:28.140455 14885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:43:28.149912 14895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:43:28.150134 14898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:43:28.149917 14896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:43:28.150203 14885 server_base.cc:1034] running on GCE node
I20260927 12:43:28.150671 14885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:43:28.151293 14885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:43:28.152499 14885 hybrid_clock.cc:648] HybridClock initialized: now 1790513008152463 us; error 46 us; skew 500 ppm
Sep 27 12:43:28 dist-test-slave-bnc1 krb5kdc[14113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513008, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:43:28.155294 14885 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 12:43:28.156118 14885 webserver.cc:460] Webserver started at http://127.0.98.2:35803/ using document root <none> and password file <none>
I20260927 12:43:28.156662 14885 fs_manager.cc:359] Metadata directory not provided
I20260927 12:43:28.156759 14885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:43:28.160565 14885 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:43:28.162683 14905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:43:28.163846 14885 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 12:43:28.163995 14885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-1/wal
uuid: "79e57a29f88f433aa4f1215486f9545d"
format_stamp: "Formatted at 2026-09-27 12:43:26 on dist-test-slave-bnc1"
server_key: "aa1e740c243c0d5bc534eca17e5e3c13"
server_key_iv: "1be7993cff0c56686ef49b7f9037e3e7"
server_key_version: "encryptionkey@0"
I20260927 12:43:28.164577 14885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:43:28.199816 14885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:43:28.203431 14885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:43:28.203663 14885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:43:28.204293 14885 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:43:28.205446 14912 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260927 12:43:28.209336 14912 ts_tablet_manager.cc:1852] T 2b048434037c4423bad26da0dde0a2e9 P 79e57a29f88f433aa4f1215486f9545d: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260927 12:43:28.209739 14912 ts_tablet_manager.cc:1905] T 2b048434037c4423bad26da0dde0a2e9 P 79e57a29f88f433aa4f1215486f9545d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:43:28.214365 14912 ts_tablet_manager.cc:1918] T 2b048434037c4423bad26da0dde0a2e9 P 79e57a29f88f433aa4f1215486f9545d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:43:28.214465 14912 log.cc:1196] T 2b048434037c4423bad26da0dde0a2e9 P 79e57a29f88f433aa4f1215486f9545d: Deleting WAL directory at /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-1/wal/wals/2b048434037c4423bad26da0dde0a2e9
I20260927 12:43:28.214828 14912 ts_tablet_manager.cc:1939] T 2b048434037c4423bad26da0dde0a2e9 P 79e57a29f88f433aa4f1215486f9545d: Deleting consensus metadata
I20260927 12:43:28.215200 14885 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260927 12:43:28.215291 14885 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.000s sys 0.000s
I20260927 12:43:28.215380 14885 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:43:28.215425 14885 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:43:28.224673 14885 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33847
I20260927 12:43:28.224700 15019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33847 every 8 connection(s)
I20260927 12:43:28.225610 14885 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:43:28.225989 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 14885
I20260927 12:43:28.226471 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:35185
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:36945
--webserver_port=44043
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 12:43:28 dist-test-slave-bnc1 krb5kdc[14113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513008, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:43:28.239118 14884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:28.227623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37555 (local address 127.0.98.62:36945)
0927 12:43:28.227795 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 12:43:28.227800 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:43:28.228811 (+ 1011us) server_negotiation.cc:407] Connection header received
0927 12:43:28.229840 (+ 1029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:43:28.229844 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:43:28.229901 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:43:28.230001 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:43:28.231209 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:28.231732 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:28.232850 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:28.233090 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:28.233198 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:28.235608 (+ 2410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:43:28.235616 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:43:28.235619 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:43:28.235643 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:43:28.237103 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:28.237618 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:28.237624 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:28.237626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:28.237706 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:28.238048 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:28.238053 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:28.238059 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:28.238227 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:43:28.238333 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 12:43:28.238799 (+ 466us) server_negotiation.cc:299] Negotiation successful
0927 12:43:28.238926 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 12:43:28.239852 15023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:28.228121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36945 (local address 127.0.98.2:37555)
0927 12:43:28.228655 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:28.228698 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 12:43:28.229551 (+ 853us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:43:28.230189 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:43:28.230196 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:43:28.230600 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:43:28.231068 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:28.231081 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:28.231853 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:28.231857 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:28.232715 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:28.232724 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:28.233232 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:28.233237 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:28.233405 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:28.234065 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:43:28.234080 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:43:28.235441 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:43:28.237218 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:43:28.237224 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:43:28.237237 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:43:28.237483 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:43:28.237805 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:43:28.237808 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:43:28.237809 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:43:28.237939 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:43:28.238330 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:43:28.238336 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:43:28.238569 (+ 233us) client_negotiation.cc:769] Sending connection context
0927 12:43:28.238742 (+ 173us) client_negotiation.cc:240] Negotiation successful
0927 12:43:28.239042 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":143,"threads_started":1}
I20260927 12:43:28.241225 15020 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36945
I20260927 12:43:28.241498 15020 heartbeater.cc:464] Registering TS with master...
I20260927 12:43:28.241986 15020 heartbeater.cc:511] Master 127.0.98.62:36945 requested a full tablet report, sending...
I20260927 12:43:28.243036 14693 ts_manager.cc:194] Registered new tserver with Master: 79e57a29f88f433aa4f1215486f9545d (127.0.98.2:33847)
I20260927 12:43:28.243767 14693 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37555
W20260927 12:43:28.338526 15022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:43:28.338799 15022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:43:28.338842 15022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:43:28.343248 15022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:43:28.343302 15022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:43:28.343402 15022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:43:28.348526 15022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:35185
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=44043
--tserver_master_addrs=127.0.98.62:36945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:43:28.349479 15022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:43:28.350473 15022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:43:28.360088 15028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:43:28.360184 15031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:43:28.360173 15022 server_base.cc:1034] running on GCE node
W20260927 12:43:28.360203 15029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:43:28.360671 15022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:43:28.361285 15022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:43:28.362438 15022 hybrid_clock.cc:648] HybridClock initialized: now 1790513008362393 us; error 55 us; skew 500 ppm
Sep 27 12:43:28 dist-test-slave-bnc1 krb5kdc[14113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513008, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:43:28.365149 15022 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 12:43:28.366060 15022 webserver.cc:460] Webserver started at http://127.0.98.3:44043/ using document root <none> and password file <none>
I20260927 12:43:28.366600 15022 fs_manager.cc:359] Metadata directory not provided
I20260927 12:43:28.366694 15022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:43:28.370834 15022 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 12:43:28.373039 15038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:43:28.374413 15022 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 12:43:28.374543 15022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-2/wal
uuid: "02d683b804bd404aa1ea624a04611726"
format_stamp: "Formatted at 2026-09-27 12:43:27 on dist-test-slave-bnc1"
server_key: "990cf133240c5c145383d0b758b2e750"
server_key_iv: "2a7b5d2c81a9e9c7f980197f03a1ab35"
server_key_version: "encryptionkey@0"
I20260927 12:43:28.375087 15022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:43:28.385676 15022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:43:28.388330 15022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:43:28.388523 15022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:43:28.389114 15022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:43:28.389952 15022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:43:28.390013 15022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:43:28.390081 15022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:43:28.390125 15022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:43:28.398653 15022 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35185
I20260927 12:43:28.398679 15151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35185 every 8 connection(s)
I20260927 12:43:28.399636 15022 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:43:28.403048 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 15022
Sep 27 12:43:28 dist-test-slave-bnc1 krb5kdc[14113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513008, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:43:28.412698 15156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:28.403706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45338 (local address 127.0.98.62:36945)
0927 12:43:28.404185 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0927 12:43:28.404190 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:43:28.404201 (+ 11us) server_negotiation.cc:407] Connection header received
0927 12:43:28.404243 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:43:28.404246 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:43:28.404280 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:43:28.404359 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:43:28.405304 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:28.406102 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:28.407305 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:28.407540 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:28.407616 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:28.409281 (+ 1665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:43:28.409285 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:43:28.409287 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:43:28.409318 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:43:28.410995 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:28.411434 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:28.411438 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:28.411440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:28.411524 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:28.411809 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:28.411814 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:28.411815 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:28.412014 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:43:28.412260 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0927 12:43:28.412382 (+ 122us) server_negotiation.cc:299] Negotiation successful
0927 12:43:28.412506 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":65,"threads_started":1}
I20260927 12:43:28.415132 14884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:28.401485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54037 (local address 127.0.98.62:36945)
0927 12:43:28.401654 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 12:43:28.401659 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:43:28.402745 (+ 1086us) server_negotiation.cc:407] Connection header received
0927 12:43:28.404190 (+ 1445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:43:28.404195 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:43:28.404246 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:43:28.404301 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:43:28.406065 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:28.406760 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:28.408247 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:28.408493 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:28.408583 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:28.411516 (+ 2933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:43:28.411519 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:43:28.411521 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:43:28.411543 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:43:28.413099 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:28.413625 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:28.413629 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:28.413630 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:28.413706 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:28.414054 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:28.414057 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:28.414059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:28.414222 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:43:28.414327 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 12:43:28.414874 (+ 547us) server_negotiation.cc:299] Negotiation successful
0927 12:43:28.414982 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 12:43:28.415947 15154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:28.401941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36945 (local address 127.0.98.3:54037)
0927 12:43:28.402574 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:28.402616 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 12:43:28.403928 (+ 1312us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:43:28.404709 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:43:28.404717 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:43:28.405236 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:43:28.405875 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:28.405892 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:28.406880 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:28.406884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:28.408075 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:28.408084 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:28.408661 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:28.408668 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:28.408902 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:28.409493 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:43:28.409530 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:43:28.411343 (+ 1813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:43:28.413214 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:43:28.413219 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:43:28.413235 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:43:28.413491 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:43:28.413806 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:43:28.413809 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:43:28.413811 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:43:28.413952 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:43:28.414327 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:43:28.414335 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:43:28.414634 (+ 299us) client_negotiation.cc:769] Sending connection context
0927 12:43:28.414818 (+ 184us) client_negotiation.cc:240] Negotiation successful
0927 12:43:28.415145 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":111,"threads_started":1}
I20260927 12:43:28.416994 15152 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36945
I20260927 12:43:28.417248 15152 heartbeater.cc:464] Registering TS with master...
I20260927 12:43:28.417760 15152 heartbeater.cc:511] Master 127.0.98.62:36945 requested a full tablet report, sending...
I20260927 12:43:28.418686 14693 ts_manager.cc:194] Registered new tserver with Master: 02d683b804bd404aa1ea624a04611726 (127.0.98.3:35185)
I20260927 12:43:28.419181 14693 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54037
I20260927 12:43:28.426123 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 12:43:28.430020 14884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:28.426709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45342 (local address 127.0.98.62:36945)
0927 12:43:28.426821 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 12:43:28.426825 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:43:28.426992 (+ 167us) server_negotiation.cc:407] Connection header received
0927 12:43:28.427120 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:43:28.427123 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:43:28.427212 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:43:28.427295 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:43:28.427490 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:28.427978 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:28.428766 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:28.428965 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:28.429045 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:28.429311 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:43:28.429741 (+ 430us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:43:28.429833 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 12:43:28.429890 (+ 57us) server_negotiation.cc:299] Negotiation successful
0927 12:43:28.429929 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 12:43:28.430634 14693 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:45342:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 12:43:28.433756 14693 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:43:28.440932 14815 tablet_service.cc:1467] Processing CreateTablet for tablet 1937734a23e4420da43bae345f4a923a (DEFAULT_TABLE table=test-table [id=51401d87e6e0474186fafc674cf9afec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:43:28.441633 14815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1937734a23e4420da43bae345f4a923a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:43:28.444561 14773 tablet_bootstrap.cc:492] T 1937734a23e4420da43bae345f4a923a P 3dc0e90d397a4c1bb3749bf9a7c50732: Bootstrap starting.
I20260927 12:43:28.445086 14888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:28.439787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35185 (local address 127.0.0.1:53450)
0927 12:43:28.439929 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:28.439944 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:43:28.440023 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:43:28.441232 (+ 1209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:43:28.441236 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:43:28.441254 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:43:28.441340 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:28.441344 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:28.442821 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:28.442827 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:28.443615 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:28.443623 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:28.444666 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:28.444669 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:28.444769 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:28.444816 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 12:43:28.444872 (+ 56us) client_negotiation.cc:240] Negotiation successful
0927 12:43:28.444963 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260927 12:43:28.445470 14773 tablet_bootstrap.cc:654] T 1937734a23e4420da43bae345f4a923a P 3dc0e90d397a4c1bb3749bf9a7c50732: Neither blocks nor log segments found. Creating new log.
I20260927 12:43:28.445822 15161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:28.439961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53450 (local address 127.0.98.3:35185)
0927 12:43:28.440736 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0927 12:43:28.440748 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 12:43:28.440771 (+ 23us) server_negotiation.cc:407] Connection header received
0927 12:43:28.440845 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:43:28.440850 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:43:28.441088 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:43:28.441253 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:43:28.441467 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:28.442709 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:28.443741 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:28.444515 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:28.445322 (+ 807us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:28.445501 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 12:43:28.445555 (+ 54us) server_negotiation.cc:299] Negotiation successful
0927 12:43:28.445686 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":664,"thread_start_us":162,"threads_started":1}
I20260927 12:43:28.446988 15162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:28.440103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33847 (local address 127.0.0.1:43688)
0927 12:43:28.441477 (+ 1374us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:28.441490 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:43:28.441612 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:43:28.442233 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:43:28.442239 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:43:28.442533 (+ 294us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:43:28.442624 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:28.442630 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:28.443988 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:28.443991 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:28.445438 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:28.445448 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:28.446544 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:28.446549 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:28.446665 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:28.446702 (+ 37us) client_negotiation.cc:769] Sending connection context
0927 12:43:28.446793 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 12:43:28.446863 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1304,"thread_start_us":96,"threads_started":1}
I20260927 12:43:28.446998 14773 tablet_bootstrap.cc:492] T 1937734a23e4420da43bae345f4a923a P 3dc0e90d397a4c1bb3749bf9a7c50732: No bootstrap required, opened a new log
I20260927 12:43:28.447050 15163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:28.440292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43688 (local address 127.0.98.2:33847)
0927 12:43:28.440598 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0927 12:43:28.440616 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 12:43:28.441649 (+ 1033us) server_negotiation.cc:407] Connection header received
0927 12:43:28.441770 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:43:28.441778 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:43:28.441976 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:43:28.442138 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:43:28.442761 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:28.443852 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:28.445598 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:28.446414 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:28.446537 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:28.446673 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 12:43:28.446820 (+ 147us) server_negotiation.cc:299] Negotiation successful
0927 12:43:28.446902 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":131,"threads_started":1}
I20260927 12:43:28.447098 14773 ts_tablet_manager.cc:1397] T 1937734a23e4420da43bae345f4a923a P 3dc0e90d397a4c1bb3749bf9a7c50732: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260927 12:43:28.447499 14773 raft_consensus.cc:348] T 1937734a23e4420da43bae345f4a923a P 3dc0e90d397a4c1bb3749bf9a7c50732 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02d683b804bd404aa1ea624a04611726" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35185 } } peers { permanent_uuid: "79e57a29f88f433aa4f1215486f9545d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33847 } } peers { permanent_uuid: "3dc0e90d397a4c1bb3749bf9a7c50732" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38451 } }
I20260927 12:43:28.447623 14773 raft_consensus.cc:374] T 1937734a23e4420da43bae345f4a923a P 3dc0e90d397a4c1bb3749bf9a7c50732 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:43:28.447671 14773 raft_consensus.cc:729] T 1937734a23e4420da43bae345f4a923a P 3dc0e90d397a4c1bb3749bf9a7c50732 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dc0e90d397a4c1bb3749bf9a7c50732, State: Initialized, Role: FOLLOWER
I20260927 12:43:28.447803 14773 consensus_queue.cc:260] T 1937734a23e4420da43bae345f4a923a P 3dc0e90d397a4c1bb3749bf9a7c50732 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02d683b804bd404aa1ea624a04611726" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35185 } } peers { permanent_uuid: "79e57a29f88f433aa4f1215486f9545d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33847 } } peers { permanent_uuid: "3dc0e90d397a4c1bb3749bf9a7c50732" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38451 } }
I20260927 12:43:28.448068 14773 ts_tablet_manager.cc:1428] T 1937734a23e4420da43bae345f4a923a P 3dc0e90d397a4c1bb3749bf9a7c50732: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 12:43:28.448764 15086 tablet_service.cc:1467] Processing CreateTablet for tablet 1937734a23e4420da43bae345f4a923a (DEFAULT_TABLE table=test-table [id=51401d87e6e0474186fafc674cf9afec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:43:28.449743 15086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1937734a23e4420da43bae345f4a923a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:43:28.450973 14954 tablet_service.cc:1467] Processing CreateTablet for tablet 1937734a23e4420da43bae345f4a923a (DEFAULT_TABLE table=test-table [id=51401d87e6e0474186fafc674cf9afec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:43:28.451865 14954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1937734a23e4420da43bae345f4a923a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:43:28.455947 14912 tablet_bootstrap.cc:492] T 1937734a23e4420da43bae345f4a923a P 79e57a29f88f433aa4f1215486f9545d: Bootstrap starting.
I20260927 12:43:28.456806 15165 tablet_bootstrap.cc:492] T 1937734a23e4420da43bae345f4a923a P 02d683b804bd404aa1ea624a04611726: Bootstrap starting.
I20260927 12:43:28.458014 14912 tablet_bootstrap.cc:654] T 1937734a23e4420da43bae345f4a923a P 79e57a29f88f433aa4f1215486f9545d: Neither blocks nor log segments found. Creating new log.
I20260927 12:43:28.458762 14912 log.cc:824] T 1937734a23e4420da43bae345f4a923a P 79e57a29f88f433aa4f1215486f9545d: Log is configured to *not* fsync() on all Append() calls
I20260927 12:43:28.459209 15165 tablet_bootstrap.cc:654] T 1937734a23e4420da43bae345f4a923a P 02d683b804bd404aa1ea624a04611726: Neither blocks nor log segments found. Creating new log.
I20260927 12:43:28.460017 15165 log.cc:824] T 1937734a23e4420da43bae345f4a923a P 02d683b804bd404aa1ea624a04611726: Log is configured to *not* fsync() on all Append() calls
I20260927 12:43:28.460821 14912 tablet_bootstrap.cc:492] T 1937734a23e4420da43bae345f4a923a P 79e57a29f88f433aa4f1215486f9545d: No bootstrap required, opened a new log
I20260927 12:43:28.461004 14912 ts_tablet_manager.cc:1397] T 1937734a23e4420da43bae345f4a923a P 79e57a29f88f433aa4f1215486f9545d: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260927 12:43:28.461782 15165 tablet_bootstrap.cc:492] T 1937734a23e4420da43bae345f4a923a P 02d683b804bd404aa1ea624a04611726: No bootstrap required, opened a new log
I20260927 12:43:28.461968 15165 ts_tablet_manager.cc:1397] T 1937734a23e4420da43bae345f4a923a P 02d683b804bd404aa1ea624a04611726: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.004s
I20260927 12:43:28.463936 14912 raft_consensus.cc:348] T 1937734a23e4420da43bae345f4a923a P 79e57a29f88f433aa4f1215486f9545d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02d683b804bd404aa1ea624a04611726" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35185 } } peers { permanent_uuid: "79e57a29f88f433aa4f1215486f9545d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33847 } } peers { permanent_uuid: "3dc0e90d397a4c1bb3749bf9a7c50732" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38451 } }
I20260927 12:43:28.464139 14912 raft_consensus.cc:374] T 1937734a23e4420da43bae345f4a923a P 79e57a29f88f433aa4f1215486f9545d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:43:28.464196 14912 raft_consensus.cc:729] T 1937734a23e4420da43bae345f4a923a P 79e57a29f88f433aa4f1215486f9545d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79e57a29f88f433aa4f1215486f9545d, State: Initialized, Role: FOLLOWER
I20260927 12:43:28.464645 14912 consensus_queue.cc:260] T 1937734a23e4420da43bae345f4a923a P 79e57a29f88f433aa4f1215486f9545d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02d683b804bd404aa1ea624a04611726" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35185 } } peers { permanent_uuid: "79e57a29f88f433aa4f1215486f9545d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33847 } } peers { permanent_uuid: "3dc0e90d397a4c1bb3749bf9a7c50732" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38451 } }
I20260927 12:43:28.464793 15165 raft_consensus.cc:348] T 1937734a23e4420da43bae345f4a923a P 02d683b804bd404aa1ea624a04611726 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02d683b804bd404aa1ea624a04611726" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35185 } } peers { permanent_uuid: "79e57a29f88f433aa4f1215486f9545d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33847 } } peers { permanent_uuid: "3dc0e90d397a4c1bb3749bf9a7c50732" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38451 } }
I20260927 12:43:28.465080 15165 raft_consensus.cc:374] T 1937734a23e4420da43bae345f4a923a P 02d683b804bd404aa1ea624a04611726 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:43:28.465149 15165 raft_consensus.cc:729] T 1937734a23e4420da43bae345f4a923a P 02d683b804bd404aa1ea624a04611726 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02d683b804bd404aa1ea624a04611726, State: Initialized, Role: FOLLOWER
I20260927 12:43:28.465351 15020 heartbeater.cc:503] Master 127.0.98.62:36945 was elected leader, sending a full tablet report...
I20260927 12:43:28.465622 15165 consensus_queue.cc:260] T 1937734a23e4420da43bae345f4a923a P 02d683b804bd404aa1ea624a04611726 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02d683b804bd404aa1ea624a04611726" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35185 } } peers { permanent_uuid: "79e57a29f88f433aa4f1215486f9545d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33847 } } peers { permanent_uuid: "3dc0e90d397a4c1bb3749bf9a7c50732" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38451 } }
I20260927 12:43:28.465936 14912 ts_tablet_manager.cc:1428] T 1937734a23e4420da43bae345f4a923a P 79e57a29f88f433aa4f1215486f9545d: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260927 12:43:28.466462 15165 ts_tablet_manager.cc:1428] T 1937734a23e4420da43bae345f4a923a P 02d683b804bd404aa1ea624a04611726: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260927 12:43:28.466482 15152 heartbeater.cc:503] Master 127.0.98.62:36945 was elected leader, sending a full tablet report...
W20260927 12:43:28.476679 15021 tablet.cc:2369] T 1937734a23e4420da43bae345f4a923a P 79e57a29f88f433aa4f1215486f9545d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 12:43:28.505463 14882 tablet.cc:2369] T 1937734a23e4420da43bae345f4a923a P 3dc0e90d397a4c1bb3749bf9a7c50732: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:43:28.509421 15169 raft_consensus.cc:482] T 1937734a23e4420da43bae345f4a923a P 02d683b804bd404aa1ea624a04611726 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:43:28.509696 15169 raft_consensus.cc:504] T 1937734a23e4420da43bae345f4a923a P 02d683b804bd404aa1ea624a04611726 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02d683b804bd404aa1ea624a04611726" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35185 } } peers { permanent_uuid: "79e57a29f88f433aa4f1215486f9545d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33847 } } peers { permanent_uuid: "3dc0e90d397a4c1bb3749bf9a7c50732" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38451 } }
I20260927 12:43:28.510802 15169 leader_election.cc:288] T 1937734a23e4420da43bae345f4a923a P 02d683b804bd404aa1ea624a04611726 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79e57a29f88f433aa4f1215486f9545d (127.0.98.2:33847), 3dc0e90d397a4c1bb3749bf9a7c50732 (127.0.98.1:38451)
I20260927 12:43:28.514309 15163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:28.511190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49181 (local address 127.0.98.2:33847)
0927 12:43:28.511314 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 12:43:28.511318 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:43:28.511332 (+ 14us) server_negotiation.cc:407] Connection header received
0927 12:43:28.511383 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:43:28.511385 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:43:28.511424 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:43:28.511529 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:43:28.511810 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:28.512299 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:28.513318 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:28.513751 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:28.513871 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:28.513935 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 12:43:28.514122 (+ 187us) server_negotiation.cc:299] Negotiation successful
0927 12:43:28.514196 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 12:43:28.514309 15154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:28.511048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33847 (local address 127.0.98.3:49181)
0927 12:43:28.511162 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:28.511179 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 12:43:28.511302 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:43:28.511543 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:43:28.511547 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:43:28.511564 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:43:28.511684 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:28.511689 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:28.512409 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:28.512415 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:28.513185 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:28.513193 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:28.513871 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:28.513875 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:28.513980 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:28.514007 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 12:43:28.514094 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 12:43:28.514178 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260927 12:43:28.514914 14974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1937734a23e4420da43bae345f4a923a" candidate_uuid: "02d683b804bd404aa1ea624a04611726" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79e57a29f88f433aa4f1215486f9545d" is_pre_election: true
I20260927 12:43:28.514994 15170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:28.511048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38451 (local address 127.0.98.3:45929)
0927 12:43:28.511681 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:28.511692 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 12:43:28.511804 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:43:28.512093 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:43:28.512096 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:43:28.512111 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:43:28.512235 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:28.512240 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:28.512966 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:28.512972 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:28.513812 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:28.513821 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:28.514585 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:28.514589 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:28.514690 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:28.514710 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 12:43:28.514791 (+ 81us) client_negotiation.cc:240] Negotiation successful
0927 12:43:28.514858 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"mutex_wait_us":330,"thread_start_us":136,"threads_started":1}
I20260927 12:43:28.514991 14889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:28.511182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45929 (local address 127.0.98.1:38451)
0927 12:43:28.511328 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 12:43:28.511335 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 12:43:28.511775 (+ 440us) server_negotiation.cc:407] Connection header received
0927 12:43:28.511918 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:43:28.511921 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:43:28.511983 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:43:28.512085 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:43:28.512345 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:28.512854 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:28.513930 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:28.514431 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:28.514548 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:28.514702 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 12:43:28.514814 (+ 112us) server_negotiation.cc:299] Negotiation successful
0927 12:43:28.514885 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 12:43:28.515270 14974 raft_consensus.cc:2454] T 1937734a23e4420da43bae345f4a923a P 79e57a29f88f433aa4f1215486f9545d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02d683b804bd404aa1ea624a04611726 in term 0.
I20260927 12:43:28.515699 14835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1937734a23e4420da43bae345f4a923a" candidate_uuid: "02d683b804bd404aa1ea624a04611726" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3dc0e90d397a4c1bb3749bf9a7c50732" is_pre_election: true
I20260927 12:43:28.515802 15042 leader_election.cc:302] T 1937734a23e4420da43bae345f4a923a P 02d683b804bd404aa1ea624a04611726 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02d683b804bd404aa1ea624a04611726, 79e57a29f88f433aa4f1215486f9545d; no voters:
I20260927 12:43:28.516010 14835 raft_consensus.cc:2454] T 1937734a23e4420da43bae345f4a923a P 3dc0e90d397a4c1bb3749bf9a7c50732 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02d683b804bd404aa1ea624a04611726 in term 0.
I20260927 12:43:28.516072 15169 raft_consensus.cc:2781] T 1937734a23e4420da43bae345f4a923a P 02d683b804bd404aa1ea624a04611726 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:43:28.516217 15169 raft_consensus.cc:482] T 1937734a23e4420da43bae345f4a923a P 02d683b804bd404aa1ea624a04611726 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:43:28.516287 15169 raft_consensus.cc:3037] T 1937734a23e4420da43bae345f4a923a P 02d683b804bd404aa1ea624a04611726 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:43:28.517397 15169 raft_consensus.cc:504] T 1937734a23e4420da43bae345f4a923a P 02d683b804bd404aa1ea624a04611726 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02d683b804bd404aa1ea624a04611726" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35185 } } peers { permanent_uuid: "79e57a29f88f433aa4f1215486f9545d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33847 } } peers { permanent_uuid: "3dc0e90d397a4c1bb3749bf9a7c50732" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38451 } }
I20260927 12:43:28.517804 15169 leader_election.cc:288] T 1937734a23e4420da43bae345f4a923a P 02d683b804bd404aa1ea624a04611726 [CANDIDATE]: Term 1 election: Requested vote from peers 79e57a29f88f433aa4f1215486f9545d (127.0.98.2:33847), 3dc0e90d397a4c1bb3749bf9a7c50732 (127.0.98.1:38451)
I20260927 12:43:28.518122 14974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1937734a23e4420da43bae345f4a923a" candidate_uuid: "02d683b804bd404aa1ea624a04611726" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79e57a29f88f433aa4f1215486f9545d"
I20260927 12:43:28.518139 14835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1937734a23e4420da43bae345f4a923a" candidate_uuid: "02d683b804bd404aa1ea624a04611726" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3dc0e90d397a4c1bb3749bf9a7c50732"
I20260927 12:43:28.518251 14835 raft_consensus.cc:3037] T 1937734a23e4420da43bae345f4a923a P 3dc0e90d397a4c1bb3749bf9a7c50732 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:43:28.518265 14974 raft_consensus.cc:3037] T 1937734a23e4420da43bae345f4a923a P 79e57a29f88f433aa4f1215486f9545d [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:43:28.519399 14974 raft_consensus.cc:2454] T 1937734a23e4420da43bae345f4a923a P 79e57a29f88f433aa4f1215486f9545d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02d683b804bd404aa1ea624a04611726 in term 1.
I20260927 12:43:28.519423 14835 raft_consensus.cc:2454] T 1937734a23e4420da43bae345f4a923a P 3dc0e90d397a4c1bb3749bf9a7c50732 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02d683b804bd404aa1ea624a04611726 in term 1.
I20260927 12:43:28.519692 15042 leader_election.cc:302] T 1937734a23e4420da43bae345f4a923a P 02d683b804bd404aa1ea624a04611726 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02d683b804bd404aa1ea624a04611726, 79e57a29f88f433aa4f1215486f9545d; no voters:
I20260927 12:43:28.519975 15169 raft_consensus.cc:2781] T 1937734a23e4420da43bae345f4a923a P 02d683b804bd404aa1ea624a04611726 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:43:28.520299 15169 raft_consensus.cc:686] T 1937734a23e4420da43bae345f4a923a P 02d683b804bd404aa1ea624a04611726 [term 1 LEADER]: Becoming Leader. State: Replica: 02d683b804bd404aa1ea624a04611726, State: Running, Role: LEADER
I20260927 12:43:28.520663 15169 consensus_queue.cc:237] T 1937734a23e4420da43bae345f4a923a P 02d683b804bd404aa1ea624a04611726 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02d683b804bd404aa1ea624a04611726" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35185 } } peers { permanent_uuid: "79e57a29f88f433aa4f1215486f9545d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33847 } } peers { permanent_uuid: "3dc0e90d397a4c1bb3749bf9a7c50732" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38451 } }
I20260927 12:43:28.529232 14693 catalog_manager.cc:5495] T 1937734a23e4420da43bae345f4a923a P 02d683b804bd404aa1ea624a04611726 reported cstate change: term changed from 0 to 1, leader changed from <none> to 02d683b804bd404aa1ea624a04611726 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "02d683b804bd404aa1ea624a04611726" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02d683b804bd404aa1ea624a04611726" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79e57a29f88f433aa4f1215486f9545d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3dc0e90d397a4c1bb3749bf9a7c50732" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38451 } health_report { overall_health: UNKNOWN } } }
I20260927 12:43:28.547290 15161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:28.543610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53454 (local address 127.0.98.3:35185)
0927 12:43:28.543751 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 12:43:28.543756 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:43:28.543769 (+ 13us) server_negotiation.cc:407] Connection header received
0927 12:43:28.543921 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:43:28.543923 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:43:28.543987 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:43:28.544094 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:43:28.544328 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:28.544852 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:28.545730 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:28.545961 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:28.546069 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:28.546380 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:43:28.546915 (+ 535us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:43:28.547030 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 12:43:28.547141 (+ 111us) server_negotiation.cc:299] Negotiation successful
0927 12:43:28.547189 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 12:43:28.552723 14835 raft_consensus.cc:1261] T 1937734a23e4420da43bae345f4a923a P 3dc0e90d397a4c1bb3749bf9a7c50732 [term 1 FOLLOWER]: Refusing update from remote peer 02d683b804bd404aa1ea624a04611726: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:43:28.552726 14974 raft_consensus.cc:1261] T 1937734a23e4420da43bae345f4a923a P 79e57a29f88f433aa4f1215486f9545d [term 1 FOLLOWER]: Refusing update from remote peer 02d683b804bd404aa1ea624a04611726: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:43:28.553375 15169 consensus_queue.cc:1035] T 1937734a23e4420da43bae345f4a923a P 02d683b804bd404aa1ea624a04611726 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79e57a29f88f433aa4f1215486f9545d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:43:28.553532 15171 consensus_queue.cc:1035] T 1937734a23e4420da43bae345f4a923a P 02d683b804bd404aa1ea624a04611726 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3dc0e90d397a4c1bb3749bf9a7c50732" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:43:28.559659 15174 mvcc.cc:205] Tried to move back new op lower bound from 7333941283025498112 to 7333941282903703552. Current Snapshot: MvccSnapshot[applied={T|T < 7333941283025498112}]
I20260927 12:43:28.561457 15175 mvcc.cc:205] Tried to move back new op lower bound from 7333941283025498112 to 7333941282903703552. Current Snapshot: MvccSnapshot[applied={T|T < 7333941283025498112}]
I20260927 12:43:28.567157 14693 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:45342:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:43:28.567293 14693 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:45342:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:43:28.569702 14693 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4027986ce39f4ac898f9f1f7b7a98a93: Sending DeleteTablet for 3 replicas of tablet 1937734a23e4420da43bae345f4a923a
I20260927 12:43:28.570389 14815 tablet_service.cc:1514] Processing DeleteTablet for tablet 1937734a23e4420da43bae345f4a923a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:43:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60046
I20260927 12:43:28.570447 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 14731
I20260927 12:43:28.570417 15086 tablet_service.cc:1514] Processing DeleteTablet for tablet 1937734a23e4420da43bae345f4a923a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:43:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53450
I20260927 12:43:28.570967 14954 tablet_service.cc:1514] Processing DeleteTablet for tablet 1937734a23e4420da43bae345f4a923a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:43:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43688
I20260927 12:43:28.571487 15183 tablet_replica.cc:330] T 1937734a23e4420da43bae345f4a923a P 79e57a29f88f433aa4f1215486f9545d: stopping tablet replica
I20260927 12:43:28.571698 15182 tablet_replica.cc:330] T 1937734a23e4420da43bae345f4a923a P 02d683b804bd404aa1ea624a04611726: stopping tablet replica
I20260927 12:43:28.571810 15183 raft_consensus.cc:2229] T 1937734a23e4420da43bae345f4a923a P 79e57a29f88f433aa4f1215486f9545d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:43:28.572109 15183 raft_consensus.cc:2258] T 1937734a23e4420da43bae345f4a923a P 79e57a29f88f433aa4f1215486f9545d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:43:28.572115 15182 raft_consensus.cc:2229] T 1937734a23e4420da43bae345f4a923a P 02d683b804bd404aa1ea624a04611726 [term 1 LEADER]: Raft consensus shutting down.
I20260927 12:43:28.572914 15182 raft_consensus.cc:2258] T 1937734a23e4420da43bae345f4a923a P 02d683b804bd404aa1ea624a04611726 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:43:28.573520 15183 ts_tablet_manager.cc:1905] T 1937734a23e4420da43bae345f4a923a P 79e57a29f88f433aa4f1215486f9545d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:43:28.574287 15182 ts_tablet_manager.cc:1905] T 1937734a23e4420da43bae345f4a923a P 02d683b804bd404aa1ea624a04611726: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:43:28.576265 15183 ts_tablet_manager.cc:1918] T 1937734a23e4420da43bae345f4a923a P 79e57a29f88f433aa4f1215486f9545d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:43:28.576399 15183 log.cc:1196] T 1937734a23e4420da43bae345f4a923a P 79e57a29f88f433aa4f1215486f9545d: Deleting WAL directory at /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-1/wal/wals/1937734a23e4420da43bae345f4a923a
I20260927 12:43:28.576746 15183 ts_tablet_manager.cc:1939] T 1937734a23e4420da43bae345f4a923a P 79e57a29f88f433aa4f1215486f9545d: Deleting consensus metadata
I20260927 12:43:28.577030 15182 ts_tablet_manager.cc:1918] T 1937734a23e4420da43bae345f4a923a P 02d683b804bd404aa1ea624a04611726: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:43:28.577186 15182 log.cc:1196] T 1937734a23e4420da43bae345f4a923a P 02d683b804bd404aa1ea624a04611726: Deleting WAL directory at /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-2/wal/wals/1937734a23e4420da43bae345f4a923a
I20260927 12:43:28.577581 15182 ts_tablet_manager.cc:1939] T 1937734a23e4420da43bae345f4a923a P 02d683b804bd404aa1ea624a04611726: Deleting consensus metadata
I20260927 12:43:28.578028 14680 catalog_manager.cc:4841] TS 79e57a29f88f433aa4f1215486f9545d (127.0.98.2:33847): tablet 1937734a23e4420da43bae345f4a923a (table test-table [id=51401d87e6e0474186fafc674cf9afec]) successfully deleted
I20260927 12:43:28.578780 14680 catalog_manager.cc:4841] TS 02d683b804bd404aa1ea624a04611726 (127.0.98.3:35185): tablet 1937734a23e4420da43bae345f4a923a (table test-table [id=51401d87e6e0474186fafc674cf9afec]) successfully deleted
W20260927 12:43:28.579231 14677 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38451 (error 108)
I20260927 12:43:28.579494 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 14885
I20260927 12:43:28.580922 15162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:28.580626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38451 (local address 127.0.0.1:60050)
0927 12:43:28.580779 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:28.580847 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 12:43:28.581135 14677 catalog_manager.cc:4584] TS 3dc0e90d397a4c1bb3749bf9a7c50732 (127.0.98.1:38451): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1937734a23e4420da43bae345f4a923a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38451: connect: Connection refused (error 111)
I20260927 12:43:28.587880 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 15022
I20260927 12:43:28.593346 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 14660
I20260927 12:43:28.599656 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36945
--webserver_interface=127.0.98.62
--webserver_port=45379
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36945
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:43:28.715570 15185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:43:28.715839 15185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:43:28.715881 15185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:43:28.720083 15185 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:43:28.720160 15185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:43:28.720180 15185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:43:28.720197 15185 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:43:28.720212 15185 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:43:28.725728 15185 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36945
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=45379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15185
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:43:28.726861 15185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:43:28.727974 15185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 12:43:28.736946 15185 server_base.cc:1034] running on GCE node
W20260927 12:43:28.736900 15190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:43:28.736886 15191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:43:28.737090 15193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:43:28.737797 15185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:43:28.738914 15185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:43:28.740072 15185 hybrid_clock.cc:648] HybridClock initialized: now 1790513008740055 us; error 27 us; skew 500 ppm
Sep 27 12:43:28 dist-test-slave-bnc1 krb5kdc[14113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513008, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:43:28.743026 15185 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:43:28.743937 15185 webserver.cc:460] Webserver started at http://127.0.98.62:45379/ using document root <none> and password file <none>
I20260927 12:43:28.744513 15185 fs_manager.cc:359] Metadata directory not provided
I20260927 12:43:28.744608 15185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:43:28.748723 15185 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 12:43:28.751098 15200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:43:28.752527 15185 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 12:43:28.752696 15185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/master-0/wal
uuid: "4027986ce39f4ac898f9f1f7b7a98a93"
format_stamp: "Formatted at 2026-09-27 12:43:26 on dist-test-slave-bnc1"
server_key: "8d9e049b3635deaa906bc249e97dca15"
server_key_iv: "f6360a5d8bbd11b7e31d6a6f64f31fd6"
server_key_version: "encryptionkey@0"
I20260927 12:43:28.753286 15185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:43:28.768390 15185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:43:28.771329 15185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:43:28.771519 15185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:43:28.778126 15185 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36945
I20260927 12:43:28.778393 15252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36945 every 8 connection(s)
I20260927 12:43:28.779170 15185 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:43:28.783592 15253 sys_catalog.cc:263] Verifying existing consensus state
I20260927 12:43:28.785183 15253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4027986ce39f4ac898f9f1f7b7a98a93: Bootstrap starting.
I20260927 12:43:28.786814 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 15185
I20260927 12:43:28.787324 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:38451
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:36945
--webserver_port=43873
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 12:43:28.803222 15253 log.cc:824] T 00000000000000000000000000000000 P 4027986ce39f4ac898f9f1f7b7a98a93: Log is configured to *not* fsync() on all Append() calls
I20260927 12:43:28.814400 15253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4027986ce39f4ac898f9f1f7b7a98a93: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 12:43:28.814776 15253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4027986ce39f4ac898f9f1f7b7a98a93: Bootstrap complete.
I20260927 12:43:28.820878 15253 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4027986ce39f4ac898f9f1f7b7a98a93 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4027986ce39f4ac898f9f1f7b7a98a93" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36945 } }
I20260927 12:43:28.821444 15253 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4027986ce39f4ac898f9f1f7b7a98a93 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4027986ce39f4ac898f9f1f7b7a98a93, State: Initialized, Role: FOLLOWER
I20260927 12:43:28.822125 15253 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4027986ce39f4ac898f9f1f7b7a98a93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4027986ce39f4ac898f9f1f7b7a98a93" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36945 } }
I20260927 12:43:28.822347 15253 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4027986ce39f4ac898f9f1f7b7a98a93 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:43:28.822441 15253 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4027986ce39f4ac898f9f1f7b7a98a93 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:43:28.822571 15253 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4027986ce39f4ac898f9f1f7b7a98a93 [term 2 FOLLOWER]: Advancing to term 3
I20260927 12:43:28.825297 15253 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4027986ce39f4ac898f9f1f7b7a98a93 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4027986ce39f4ac898f9f1f7b7a98a93" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36945 } }
I20260927 12:43:28.825740 15253 leader_election.cc:302] T 00000000000000000000000000000000 P 4027986ce39f4ac898f9f1f7b7a98a93 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4027986ce39f4ac898f9f1f7b7a98a93; no voters:
I20260927 12:43:28.826144 15253 leader_election.cc:288] T 00000000000000000000000000000000 P 4027986ce39f4ac898f9f1f7b7a98a93 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 12:43:28.826279 15257 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4027986ce39f4ac898f9f1f7b7a98a93 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 12:43:28.826627 15257 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4027986ce39f4ac898f9f1f7b7a98a93 [term 3 LEADER]: Becoming Leader. State: Replica: 4027986ce39f4ac898f9f1f7b7a98a93, State: Running, Role: LEADER
I20260927 12:43:28.826997 15257 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4027986ce39f4ac898f9f1f7b7a98a93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4027986ce39f4ac898f9f1f7b7a98a93" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36945 } }
I20260927 12:43:28.827452 15253 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4027986ce39f4ac898f9f1f7b7a98a93 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:43:28.828127 15257 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4027986ce39f4ac898f9f1f7b7a98a93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4027986ce39f4ac898f9f1f7b7a98a93. Latest consensus state: current_term: 3 leader_uuid: "4027986ce39f4ac898f9f1f7b7a98a93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4027986ce39f4ac898f9f1f7b7a98a93" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36945 } } }
I20260927 12:43:28.828343 15257 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4027986ce39f4ac898f9f1f7b7a98a93 [sys.catalog]: This master's current role is: LEADER
I20260927 12:43:28.828526 15258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4027986ce39f4ac898f9f1f7b7a98a93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "4027986ce39f4ac898f9f1f7b7a98a93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4027986ce39f4ac898f9f1f7b7a98a93" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36945 } } }
I20260927 12:43:28.828697 15258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4027986ce39f4ac898f9f1f7b7a98a93 [sys.catalog]: This master's current role is: LEADER
I20260927 12:43:28.828800 15266 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:43:28.836129 15266 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:43:28.837307 15266 catalog_manager.cc:1259] Loaded cluster ID: 102886e1aa324ab1bf5890f54a30055d
I20260927 12:43:28.837415 15266 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260927 12:43:28.838456 15266 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fe44eeb39ad google::LogMessage::Fail() at ??:0
@ 0x7fe44eeb688a google::LogMessage::SendToLog() at ??:0
@ 0x7fe44eeb34de google::LogMessage::Flush() at ??:0
@ 0x7fe44eeb4a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fe4597f2fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fe4597f3d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fe4597f0579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fe459834640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fe4511053d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fe451105c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fe4511073f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fe4510f6512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fe45273a6db start_thread at ??:0
@ 0x7fe44d9b161f clone at ??:0
W20260927 12:43:28.945325 15255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:43:28.945827 15255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:43:28.945930 15255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:43:28.952485 15255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:43:28.952584 15255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:43:28.952678 15255 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:43:28.960731 15255 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:38451
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=43873
--tserver_master_addrs=127.0.98.62:36945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15255
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:43:28.962189 15255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:43:28.963708 15255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:43:28.976136 15294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:43:28.976202 15297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:43:28.976109 15295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:43:28.976425 15255 server_base.cc:1034] running on GCE node
I20260927 12:43:28.976887 15255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:43:28.977733 15255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:43:28.978936 15255 hybrid_clock.cc:648] HybridClock initialized: now 1790513008978899 us; error 52 us; skew 500 ppm
Sep 27 12:43:28 dist-test-slave-bnc1 krb5kdc[14113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513008, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:43:28.982575 15255 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 12:43:28.983479 15255 webserver.cc:460] Webserver started at http://127.0.98.1:43873/ using document root <none> and password file <none>
I20260927 12:43:28.984156 15255 fs_manager.cc:359] Metadata directory not provided
I20260927 12:43:28.984222 15255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:43:28.988184 15255 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 12:43:28.990521 15304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:43:28.991858 15255 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:43:28.992012 15255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-0/wal
uuid: "3dc0e90d397a4c1bb3749bf9a7c50732"
format_stamp: "Formatted at 2026-09-27 12:43:26 on dist-test-slave-bnc1"
server_key: "4b83bf9be58a116b974810c085611cce"
server_key_iv: "3532482f32615bc41ad994ad71f535f9"
server_key_version: "encryptionkey@0"
I20260927 12:43:28.992585 15255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:43:29.010409 15255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:43:29.013854 15255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:43:29.014058 15255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:43:29.014726 15255 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:43:29.015832 15311 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 12:43:29.021644 15255 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 12:43:29.021723 15255 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260927 12:43:29.021807 15255 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 12:43:29.023443 15255 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 12:43:29.023511 15255 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260927 12:43:29.023818 15311 tablet_bootstrap.cc:492] T 1937734a23e4420da43bae345f4a923a P 3dc0e90d397a4c1bb3749bf9a7c50732: Bootstrap starting.
I20260927 12:43:29.034457 15255 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38451
I20260927 12:43:29.035727 15255 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:43:29.036741 15418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38451 every 8 connection(s)
I20260927 12:43:29.043607 15311 log.cc:824] T 1937734a23e4420da43bae345f4a923a P 3dc0e90d397a4c1bb3749bf9a7c50732: Log is configured to *not* fsync() on all Append() calls
I20260927 12:43:29.045097 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 15255
I20260927 12:43:29.045612 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:33847
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:36945
--webserver_port=35803
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 12:43:29.051862 15311 tablet_bootstrap.cc:492] T 1937734a23e4420da43bae345f4a923a P 3dc0e90d397a4c1bb3749bf9a7c50732: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 12:43:29.052263 15311 tablet_bootstrap.cc:492] T 1937734a23e4420da43bae345f4a923a P 3dc0e90d397a4c1bb3749bf9a7c50732: Bootstrap complete.
I20260927 12:43:29.053387 15311 ts_tablet_manager.cc:1397] T 1937734a23e4420da43bae345f4a923a P 3dc0e90d397a4c1bb3749bf9a7c50732: Time spent bootstrapping tablet: real 0.030s user 0.014s sys 0.012s
I20260927 12:43:29.055915 15311 raft_consensus.cc:348] T 1937734a23e4420da43bae345f4a923a P 3dc0e90d397a4c1bb3749bf9a7c50732 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02d683b804bd404aa1ea624a04611726" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35185 } } peers { permanent_uuid: "79e57a29f88f433aa4f1215486f9545d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33847 } } peers { permanent_uuid: "3dc0e90d397a4c1bb3749bf9a7c50732" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38451 } }
I20260927 12:43:29.056363 15311 raft_consensus.cc:729] T 1937734a23e4420da43bae345f4a923a P 3dc0e90d397a4c1bb3749bf9a7c50732 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dc0e90d397a4c1bb3749bf9a7c50732, State: Initialized, Role: FOLLOWER
I20260927 12:43:29.056885 15311 consensus_queue.cc:260] T 1937734a23e4420da43bae345f4a923a P 3dc0e90d397a4c1bb3749bf9a7c50732 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02d683b804bd404aa1ea624a04611726" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35185 } } peers { permanent_uuid: "79e57a29f88f433aa4f1215486f9545d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33847 } } peers { permanent_uuid: "3dc0e90d397a4c1bb3749bf9a7c50732" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38451 } }
I20260927 12:43:29.057965 15311 ts_tablet_manager.cc:1428] T 1937734a23e4420da43bae345f4a923a P 3dc0e90d397a4c1bb3749bf9a7c50732: Time spent starting tablet: real 0.004s user 0.007s sys 0.001s
W20260927 12:43:29.116966 15421 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:43:29.038210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36945 (local address 127.0.98.1:50657)
0927 12:43:29.038852 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:29.038936 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0927 12:43:29.040038 (+ 1102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:43:29.116225 (+ 76187us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36945: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":158,"threads_started":1}
W20260927 12:43:29.117676 15419 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36945 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:36945: Client connection negotiation failed: client connection to 127.0.98.62:36945: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 12:43:29.118430 15421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:29.118188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36945 (local address 127.0.98.1:57553)
0927 12:43:29.118315 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:29.118376 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 12:43:29.119045 15421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:29.118842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36945 (local address 127.0.98.1:47991)
0927 12:43:29.118944 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:29.118990 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 12:43:29.119208 15419 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 12:43:29.179776 15423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:43:29.180054 15423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:43:29.180152 15423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:43:29.184218 15423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:43:29.184293 15423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:43:29.184372 15423 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:43:29.189404 15423 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:33847
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=35803
--tserver_master_addrs=127.0.98.62:36945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15423
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:43:29.190398 15423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:43:29.191373 15423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:43:29.200915 15432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:43:29.200953 15429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:43:29.201108 15423 server_base.cc:1034] running on GCE node
W20260927 12:43:29.200915 15430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:43:29.201637 15423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:43:29.202250 15423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:43:29.203462 15423 hybrid_clock.cc:648] HybridClock initialized: now 1790513009203429 us; error 42 us; skew 500 ppm
Sep 27 12:43:29 dist-test-slave-bnc1 krb5kdc[14113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513009, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:43:29.206336 15423 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 12:43:29.207175 15423 webserver.cc:460] Webserver started at http://127.0.98.2:35803/ using document root <none> and password file <none>
I20260927 12:43:29.207739 15423 fs_manager.cc:359] Metadata directory not provided
I20260927 12:43:29.207839 15423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:43:29.211661 15423 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.001s
I20260927 12:43:29.213824 15439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:43:29.215029 15423 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260927 12:43:29.215170 15423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-1/wal
uuid: "79e57a29f88f433aa4f1215486f9545d"
format_stamp: "Formatted at 2026-09-27 12:43:26 on dist-test-slave-bnc1"
server_key: "aa1e740c243c0d5bc534eca17e5e3c13"
server_key_iv: "1be7993cff0c56686ef49b7f9037e3e7"
server_key_version: "encryptionkey@0"
I20260927 12:43:29.215720 15423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:43:29.240037 15423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:43:29.243671 15423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:43:29.243922 15423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:43:29.244560 15423 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:43:29.245442 15423 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:43:29.245530 15423 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:43:29.245664 15423 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:43:29.245725 15423 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:43:29.255028 15423 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33847
I20260927 12:43:29.255074 15552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33847 every 8 connection(s)
I20260927 12:43:29.256012 15423 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:43:29.259724 15555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:29.258054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36945 (local address 127.0.98.2:48749)
0927 12:43:29.258945 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:29.259050 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":691,"thread_start_us":534,"threads_started":1}
W20260927 12:43:29.260138 15553 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36945 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:36945: Client connection negotiation failed: client connection to 127.0.98.62:36945: connect: Connection refused (error 111)
I20260927 12:43:29.260696 15555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:29.260475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36945 (local address 127.0.98.2:44277)
0927 12:43:29.260586 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:29.260634 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 12:43:29.261307 15555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:29.261108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36945 (local address 127.0.98.2:42073)
0927 12:43:29.261210 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:29.261253 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 12:43:29.261487 15553 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 12:43:29.264400 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 15423
I20260927 12:43:29.264734 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:35185
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:36945
--webserver_port=44043
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:43:29.373769 15556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:43:29.374033 15556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:43:29.374078 15556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:43:29.378239 15556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:43:29.378307 15556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:43:29.378369 15556 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:43:29.383795 15556 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:35185
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=44043
--tserver_master_addrs=127.0.98.62:36945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15556
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:43:29.384886 15556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:43:29.385978 15556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:43:29.395640 15561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:43:29.395658 15564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:43:29.395658 15562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:43:29.395872 15556 server_base.cc:1034] running on GCE node
I20260927 12:43:29.396391 15556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:43:29.397040 15556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:43:29.398231 15556 hybrid_clock.cc:648] HybridClock initialized: now 1790513009398224 us; error 46 us; skew 500 ppm
Sep 27 12:43:29 dist-test-slave-bnc1 krb5kdc[14113](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513009, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:43:29.401240 15556 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 12:43:29.402217 15556 webserver.cc:460] Webserver started at http://127.0.98.3:44043/ using document root <none> and password file <none>
I20260927 12:43:29.402788 15556 fs_manager.cc:359] Metadata directory not provided
I20260927 12:43:29.402881 15556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:43:29.406798 15556 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 12:43:29.409895 15571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:43:29.411243 15556 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:43:29.411379 15556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-2/wal
uuid: "02d683b804bd404aa1ea624a04611726"
format_stamp: "Formatted at 2026-09-27 12:43:27 on dist-test-slave-bnc1"
server_key: "990cf133240c5c145383d0b758b2e750"
server_key_iv: "2a7b5d2c81a9e9c7f980197f03a1ab35"
server_key_version: "encryptionkey@0"
I20260927 12:43:29.411980 15556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:43:29.425407 15556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:43:29.428220 15556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:43:29.428421 15556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:43:29.429003 15556 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:43:29.429891 15556 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:43:29.429960 15556 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:43:29.430032 15556 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:43:29.430087 15556 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:43:29.438628 15556 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35185
I20260927 12:43:29.438692 15684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35185 every 8 connection(s)
I20260927 12:43:29.439640 15556 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:43:29.440738 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 15556
I20260927 12:43:29.443315 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:29.441867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36945 (local address 127.0.98.3:56433)
0927 12:43:29.442502 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:29.442604 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":107,"threads_started":1}
W20260927 12:43:29.443773 15685 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36945 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:36945: Client connection negotiation failed: client connection to 127.0.98.62:36945: connect: Connection refused (error 111)
I20260927 12:43:29.444367 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:29.444162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36945 (local address 127.0.98.3:37981)
0927 12:43:29.444266 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:29.444310 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 12:43:29.445026 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:29.444841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36945 (local address 127.0.98.3:37633)
0927 12:43:29.444938 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:29.444979 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 12:43:29.445240 15685 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 12:43:30.120167 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:30.119739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36945 (local address 127.0.98.1:47939)
0927 12:43:30.119976 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:30.120031 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":70,"threads_started":1}
I20260927 12:43:30.262401 15690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:30.262005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36945 (local address 127.0.98.2:50573)
0927 12:43:30.262252 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:30.262306 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":74,"threads_started":1}
I20260927 12:43:30.446208 15691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:30.445812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36945 (local address 127.0.98.3:59441)
0927 12:43:30.446035 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:30.446100 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":65,"threads_started":1}
I20260927 12:43:30.535125 15692 raft_consensus.cc:482] T 1937734a23e4420da43bae345f4a923a P 3dc0e90d397a4c1bb3749bf9a7c50732 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:43:30.535324 15692 raft_consensus.cc:504] T 1937734a23e4420da43bae345f4a923a P 3dc0e90d397a4c1bb3749bf9a7c50732 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02d683b804bd404aa1ea624a04611726" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35185 } } peers { permanent_uuid: "79e57a29f88f433aa4f1215486f9545d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33847 } } peers { permanent_uuid: "3dc0e90d397a4c1bb3749bf9a7c50732" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38451 } }
I20260927 12:43:30.536341 15692 leader_election.cc:288] T 1937734a23e4420da43bae345f4a923a P 3dc0e90d397a4c1bb3749bf9a7c50732 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02d683b804bd404aa1ea624a04611726 (127.0.98.3:35185), 79e57a29f88f433aa4f1215486f9545d (127.0.98.2:33847)
Sep 27 12:43:30 dist-test-slave-bnc1 krb5kdc[14113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513008, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 27 12:43:30 dist-test-slave-bnc1 krb5kdc[14113](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513008, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260927 12:43:30.550635 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:30.536566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35185 (local address 127.0.98.1:39345)
0927 12:43:30.536725 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:30.536742 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 12:43:30.536860 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:43:30.538702 (+ 1842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:43:30.538708 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:43:30.539255 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:43:30.539692 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:30.539706 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:30.541145 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:30.541149 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:30.542084 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:30.542095 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:30.543207 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:30.543211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:30.543323 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:30.544031 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:43:30.544042 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:43:30.545528 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:43:30.547841 (+ 2313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:43:30.547849 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:43:30.547863 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:43:30.548192 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:43:30.548618 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:43:30.548620 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:43:30.548621 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:43:30.548832 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:43:30.549646 (+ 814us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:43:30.549652 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:43:30.549909 (+ 257us) client_negotiation.cc:769] Sending connection context
0927 12:43:30.550096 (+ 187us) client_negotiation.cc:240] Negotiation successful
0927 12:43:30.550449 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63,"mutex_wait_us":87,"spinlock_wait_cycles":33280}
I20260927 12:43:30.550652 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:30.537103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60267 (local address 127.0.98.2:33847)
0927 12:43:30.537402 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0927 12:43:30.537429 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0927 12:43:30.537459 (+ 30us) server_negotiation.cc:407] Connection header received
0927 12:43:30.538145 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:43:30.538151 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:43:30.538439 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:43:30.538727 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:43:30.539867 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:30.541030 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:30.542268 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:30.543054 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:30.543222 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:30.546203 (+ 2981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:43:30.546210 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:43:30.546225 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:43:30.546259 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:43:30.548198 (+ 1939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:30.548709 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:30.548714 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:30.548720 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:30.548839 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:30.549203 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:30.549206 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:30.549208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:30.549650 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:43:30.549785 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 12:43:30.550211 (+ 426us) server_negotiation.cc:299] Negotiation successful
0927 12:43:30.550462 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":125,"threads_started":1}
I20260927 12:43:30.550738 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:30.536739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39345 (local address 127.0.98.3:35185)
0927 12:43:30.537114 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0927 12:43:30.537139 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0927 12:43:30.537186 (+ 47us) server_negotiation.cc:407] Connection header received
0927 12:43:30.537991 (+ 805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:43:30.537997 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:43:30.538344 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:43:30.538647 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:43:30.539845 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:30.540973 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:30.542222 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:30.543074 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:30.543229 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:30.545816 (+ 2587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:43:30.545824 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:43:30.545842 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:43:30.545883 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:43:30.547716 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:30.548344 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:30.548350 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:30.548355 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:30.548508 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:30.548941 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:30.548944 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:30.548946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:30.549475 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:43:30.549638 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 12:43:30.550248 (+ 610us) server_negotiation.cc:299] Negotiation successful
0927 12:43:30.550546 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":113,"threads_started":1}
I20260927 12:43:30.550763 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:30.536904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33847 (local address 127.0.98.1:60267)
0927 12:43:30.537148 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:30.537161 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:43:30.537286 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:43:30.538703 (+ 1417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:43:30.538709 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:43:30.539249 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:43:30.539690 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:30.539707 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:30.541216 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:30.541218 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:30.542122 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:30.542140 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:30.543160 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:30.543164 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:30.543321 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:30.543971 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:43:30.543995 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:43:30.545988 (+ 1993us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:43:30.548330 (+ 2342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:43:30.548334 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:43:30.548336 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:43:30.548598 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:43:30.548941 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:43:30.548944 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:43:30.548946 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:43:30.549054 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:43:30.549762 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:43:30.549764 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:43:30.549907 (+ 143us) client_negotiation.cc:769] Sending connection context
0927 12:43:30.550096 (+ 189us) client_negotiation.cc:240] Negotiation successful
0927 12:43:30.550546 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":241,"spinlock_wait_cycles":3328,"thread_start_us":71,"threads_started":1}
W20260927 12:43:30.553359 15308 leader_election.cc:341] T 1937734a23e4420da43bae345f4a923a P 3dc0e90d397a4c1bb3749bf9a7c50732 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 02d683b804bd404aa1ea624a04611726 (127.0.98.3:35185): Not found: Tablet not found: 1937734a23e4420da43bae345f4a923a
W20260927 12:43:30.553609 15308 leader_election.cc:341] T 1937734a23e4420da43bae345f4a923a P 3dc0e90d397a4c1bb3749bf9a7c50732 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 79e57a29f88f433aa4f1215486f9545d (127.0.98.2:33847): Not found: Tablet not found: 1937734a23e4420da43bae345f4a923a
I20260927 12:43:30.553678 15308 leader_election.cc:302] T 1937734a23e4420da43bae345f4a923a P 3dc0e90d397a4c1bb3749bf9a7c50732 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3dc0e90d397a4c1bb3749bf9a7c50732; no voters: 02d683b804bd404aa1ea624a04611726, 79e57a29f88f433aa4f1215486f9545d
I20260927 12:43:30.553948 15692 raft_consensus.cc:2726] T 1937734a23e4420da43bae345f4a923a P 3dc0e90d397a4c1bb3749bf9a7c50732 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 12:43:31.121301 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:31.120913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36945 (local address 127.0.98.1:41697)
0927 12:43:31.121154 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:31.121207 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":72,"threads_started":1}
I20260927 12:43:31.263453 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:31.263066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36945 (local address 127.0.98.2:41277)
0927 12:43:31.263298 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:31.263350 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":73,"threads_started":1}
I20260927 12:43:31.447317 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:31.446888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36945 (local address 127.0.98.3:44739)
0927 12:43:31.447167 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:31.447226 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":74,"threads_started":1}
I20260927 12:43:32.087685 15699 raft_consensus.cc:482] T 1937734a23e4420da43bae345f4a923a P 3dc0e90d397a4c1bb3749bf9a7c50732 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:43:32.087826 15699 raft_consensus.cc:504] T 1937734a23e4420da43bae345f4a923a P 3dc0e90d397a4c1bb3749bf9a7c50732 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02d683b804bd404aa1ea624a04611726" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35185 } } peers { permanent_uuid: "79e57a29f88f433aa4f1215486f9545d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33847 } } peers { permanent_uuid: "3dc0e90d397a4c1bb3749bf9a7c50732" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38451 } }
I20260927 12:43:32.088369 15699 leader_election.cc:288] T 1937734a23e4420da43bae345f4a923a P 3dc0e90d397a4c1bb3749bf9a7c50732 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02d683b804bd404aa1ea624a04611726 (127.0.98.3:35185), 79e57a29f88f433aa4f1215486f9545d (127.0.98.2:33847)
W20260927 12:43:32.089175 15308 leader_election.cc:341] T 1937734a23e4420da43bae345f4a923a P 3dc0e90d397a4c1bb3749bf9a7c50732 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 02d683b804bd404aa1ea624a04611726 (127.0.98.3:35185): Not found: Tablet not found: 1937734a23e4420da43bae345f4a923a
W20260927 12:43:32.089354 15308 leader_election.cc:341] T 1937734a23e4420da43bae345f4a923a P 3dc0e90d397a4c1bb3749bf9a7c50732 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 79e57a29f88f433aa4f1215486f9545d (127.0.98.2:33847): Not found: Tablet not found: 1937734a23e4420da43bae345f4a923a
I20260927 12:43:32.089387 15308 leader_election.cc:302] T 1937734a23e4420da43bae345f4a923a P 3dc0e90d397a4c1bb3749bf9a7c50732 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3dc0e90d397a4c1bb3749bf9a7c50732; no voters: 02d683b804bd404aa1ea624a04611726, 79e57a29f88f433aa4f1215486f9545d
I20260927 12:43:32.089653 15699 raft_consensus.cc:2726] T 1937734a23e4420da43bae345f4a923a P 3dc0e90d397a4c1bb3749bf9a7c50732 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 12:43:32.122574 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:32.122093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36945 (local address 127.0.98.1:38113)
0927 12:43:32.122400 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:32.122469 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":81,"threads_started":1}
I20260927 12:43:32.264571 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:32.264180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36945 (local address 127.0.98.2:47377)
0927 12:43:32.264428 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:32.264481 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":91,"threads_started":1}
I20260927 12:43:32.448391 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:32.448009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36945 (local address 127.0.98.3:41183)
0927 12:43:32.448242 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:32.448293 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":69,"threads_started":1}
I20260927 12:43:33.123821 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:33.123377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36945 (local address 127.0.98.1:48503)
0927 12:43:33.123650 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:33.123721 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":77,"threads_started":1}
I20260927 12:43:33.265970 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:33.265374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36945 (local address 127.0.98.2:40273)
0927 12:43:33.265724 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:33.265829 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":75,"threads_started":1}
I20260927 12:43:33.449540 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:33.449134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36945 (local address 127.0.98.3:57521)
0927 12:43:33.449397 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:33.449450 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":89,"threads_started":1}
I20260927 12:43:34.125277 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:34.124822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36945 (local address 127.0.98.1:45785)
0927 12:43:34.125118 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:34.125180 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":72,"threads_started":1}
I20260927 12:43:34.267169 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:34.266763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36945 (local address 127.0.98.2:48077)
0927 12:43:34.267018 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:34.267074 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":77,"threads_started":1}
I20260927 12:43:34.450800 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:34.450366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36945 (local address 127.0.98.3:44185)
0927 12:43:34.450647 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:34.450706 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":85,"threads_started":1}
I20260927 12:43:35.126581 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:35.126163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36945 (local address 127.0.98.1:52171)
0927 12:43:35.126439 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:35.126491 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":106,"threads_started":1}
I20260927 12:43:35.190382 15710 raft_consensus.cc:482] T 1937734a23e4420da43bae345f4a923a P 3dc0e90d397a4c1bb3749bf9a7c50732 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:43:35.190508 15710 raft_consensus.cc:504] T 1937734a23e4420da43bae345f4a923a P 3dc0e90d397a4c1bb3749bf9a7c50732 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02d683b804bd404aa1ea624a04611726" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35185 } } peers { permanent_uuid: "79e57a29f88f433aa4f1215486f9545d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33847 } } peers { permanent_uuid: "3dc0e90d397a4c1bb3749bf9a7c50732" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38451 } }
I20260927 12:43:35.191123 15710 leader_election.cc:288] T 1937734a23e4420da43bae345f4a923a P 3dc0e90d397a4c1bb3749bf9a7c50732 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02d683b804bd404aa1ea624a04611726 (127.0.98.3:35185), 79e57a29f88f433aa4f1215486f9545d (127.0.98.2:33847)
W20260927 12:43:35.191895 15308 leader_election.cc:341] T 1937734a23e4420da43bae345f4a923a P 3dc0e90d397a4c1bb3749bf9a7c50732 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 79e57a29f88f433aa4f1215486f9545d (127.0.98.2:33847): Not found: Tablet not found: 1937734a23e4420da43bae345f4a923a
W20260927 12:43:35.192029 15308 leader_election.cc:341] T 1937734a23e4420da43bae345f4a923a P 3dc0e90d397a4c1bb3749bf9a7c50732 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 02d683b804bd404aa1ea624a04611726 (127.0.98.3:35185): Not found: Tablet not found: 1937734a23e4420da43bae345f4a923a
I20260927 12:43:35.192083 15308 leader_election.cc:302] T 1937734a23e4420da43bae345f4a923a P 3dc0e90d397a4c1bb3749bf9a7c50732 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3dc0e90d397a4c1bb3749bf9a7c50732; no voters: 02d683b804bd404aa1ea624a04611726, 79e57a29f88f433aa4f1215486f9545d
I20260927 12:43:35.192247 15710 raft_consensus.cc:2726] T 1937734a23e4420da43bae345f4a923a P 3dc0e90d397a4c1bb3749bf9a7c50732 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 12:43:35.268304 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:35.267879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36945 (local address 127.0.98.2:50977)
0927 12:43:35.268157 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:35.268212 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":87,"threads_started":1}
I20260927 12:43:35.451855 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:35.451468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36945 (local address 127.0.98.3:41419)
0927 12:43:35.451720 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:35.451771 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":85,"threads_started":1}
I20260927 12:43:36.127703 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:36.127293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36945 (local address 127.0.98.1:45709)
0927 12:43:36.127551 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:36.127607 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":71,"threads_started":1}
I20260927 12:43:36.269479 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:36.269003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36945 (local address 127.0.98.2:47693)
0927 12:43:36.269334 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:36.269389 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":71,"threads_started":1}
I20260927 12:43:36.452983 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:36.452555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36945 (local address 127.0.98.3:59357)
0927 12:43:36.452821 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:36.452892 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":67,"threads_started":1}
I20260927 12:43:37.128963 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:37.128554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36945 (local address 127.0.98.1:35215)
0927 12:43:37.128801 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:37.128861 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":75,"threads_started":1}
I20260927 12:43:37.270696 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:37.270295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36945 (local address 127.0.98.2:51321)
0927 12:43:37.270543 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:37.270600 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":75,"threads_started":1}
I20260927 12:43:37.454222 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:37.453810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36945 (local address 127.0.98.3:50391)
0927 12:43:37.454061 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:37.454121 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":71,"threads_started":1}
I20260927 12:43:38.130210 15719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:38.129808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36945 (local address 127.0.98.1:47703)
0927 12:43:38.130055 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:38.130113 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":74,"threads_started":1}
I20260927 12:43:38.271868 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:38.271453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36945 (local address 127.0.98.2:35187)
0927 12:43:38.271710 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:38.271772 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":89,"threads_started":1}
I20260927 12:43:38.455447 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:38.454973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36945 (local address 127.0.98.3:57195)
0927 12:43:38.455258 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:38.455311 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":86,"threads_started":1}
I20260927 12:43:39.131517 15722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:39.131100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36945 (local address 127.0.98.1:40173)
0927 12:43:39.131343 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:39.131401 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":71,"threads_started":1}
I20260927 12:43:39.273075 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:39.272626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36945 (local address 127.0.98.2:47579)
0927 12:43:39.272896 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:39.272967 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":72,"threads_started":1}
I20260927 12:43:39.456642 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:39.456239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36945 (local address 127.0.98.3:57755)
0927 12:43:39.456500 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:39.456552 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":94,"threads_started":1}
I20260927 12:43:39.743220 15725 raft_consensus.cc:482] T 1937734a23e4420da43bae345f4a923a P 3dc0e90d397a4c1bb3749bf9a7c50732 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:43:39.743355 15725 raft_consensus.cc:504] T 1937734a23e4420da43bae345f4a923a P 3dc0e90d397a4c1bb3749bf9a7c50732 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02d683b804bd404aa1ea624a04611726" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35185 } } peers { permanent_uuid: "79e57a29f88f433aa4f1215486f9545d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33847 } } peers { permanent_uuid: "3dc0e90d397a4c1bb3749bf9a7c50732" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38451 } }
I20260927 12:43:39.743713 15725 leader_election.cc:288] T 1937734a23e4420da43bae345f4a923a P 3dc0e90d397a4c1bb3749bf9a7c50732 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 02d683b804bd404aa1ea624a04611726 (127.0.98.3:35185), 79e57a29f88f433aa4f1215486f9545d (127.0.98.2:33847)
W20260927 12:43:39.744632 15308 leader_election.cc:341] T 1937734a23e4420da43bae345f4a923a P 3dc0e90d397a4c1bb3749bf9a7c50732 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 02d683b804bd404aa1ea624a04611726 (127.0.98.3:35185): Not found: Tablet not found: 1937734a23e4420da43bae345f4a923a
W20260927 12:43:39.744827 15308 leader_election.cc:341] T 1937734a23e4420da43bae345f4a923a P 3dc0e90d397a4c1bb3749bf9a7c50732 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 79e57a29f88f433aa4f1215486f9545d (127.0.98.2:33847): Not found: Tablet not found: 1937734a23e4420da43bae345f4a923a
I20260927 12:43:39.744903 15308 leader_election.cc:302] T 1937734a23e4420da43bae345f4a923a P 3dc0e90d397a4c1bb3749bf9a7c50732 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3dc0e90d397a4c1bb3749bf9a7c50732; no voters: 02d683b804bd404aa1ea624a04611726, 79e57a29f88f433aa4f1215486f9545d
I20260927 12:43:39.745070 15725 raft_consensus.cc:2726] T 1937734a23e4420da43bae345f4a923a P 3dc0e90d397a4c1bb3749bf9a7c50732 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 12:43:40.132737 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:40.132323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36945 (local address 127.0.98.1:34595)
0927 12:43:40.132576 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:40.132644 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":85,"threads_started":1}
I20260927 12:43:40.274302 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:40.273854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36945 (local address 127.0.98.2:51017)
0927 12:43:40.274155 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:40.274209 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":89,"threads_started":1}
I20260927 12:43:40.457891 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:40.457410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36945 (local address 127.0.98.3:51995)
0927 12:43:40.457736 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:40.457793 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":87,"threads_started":1}
I20260927 12:43:41.134008 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:41.133538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36945 (local address 127.0.98.1:43437)
0927 12:43:41.133837 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:41.133903 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":85,"threads_started":1}
I20260927 12:43:41.275501 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:41.275100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36945 (local address 127.0.98.2:52733)
0927 12:43:41.275353 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:41.275407 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":94,"threads_started":1}
I20260927 12:43:41.459029 15731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:41.458620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36945 (local address 127.0.98.3:54353)
0927 12:43:41.458863 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:41.458919 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":72,"threads_started":1}
I20260927 12:43:42.135440 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:42.134959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36945 (local address 127.0.98.1:56031)
0927 12:43:42.135237 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:42.135307 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":81,"threads_started":1}
I20260927 12:43:42.276872 15733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:42.276458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36945 (local address 127.0.98.2:60615)
0927 12:43:42.276728 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:42.276782 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":99,"threads_started":1}
I20260927 12:43:42.460196 15734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:42.459787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36945 (local address 127.0.98.3:34129)
0927 12:43:42.460043 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:42.460103 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":72,"threads_started":1}
I20260927 12:43:43.136734 15735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:43.136314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36945 (local address 127.0.98.1:51579)
0927 12:43:43.136574 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:43.136636 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":77,"threads_started":1}
I20260927 12:43:43.278260 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:43.277708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36945 (local address 127.0.98.2:58123)
0927 12:43:43.278041 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:43.278116 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":74,"threads_started":1}
I20260927 12:43:43.461686 15737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:43.461115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36945 (local address 127.0.98.3:49959)
0927 12:43:43.461456 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:43.461537 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":121,"threads_started":1}
I20260927 12:43:44.138087 15738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:44.137705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36945 (local address 127.0.98.1:50233)
0927 12:43:44.137933 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:44.137986 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":67,"threads_started":1}
I20260927 12:43:44.279618 15739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:44.279181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36945 (local address 127.0.98.2:59057)
0927 12:43:44.279434 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:44.279525 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":85,"threads_started":1}
I20260927 12:43:44.450034 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 15255
I20260927 12:43:44.455994 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 15423
I20260927 12:43:44.461622 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 15556
2026-09-27T12:43:44Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19434 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:43:44 dist-test-slave-bnc1 krb5kdc[15742](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:43:44 dist-test-slave-bnc1 krb5kdc[15742](info): set up 2 sockets
Sep 27 12:43:44 dist-test-slave-bnc1 krb5kdc[15742](info): commencing operation
krb5kdc: starting...
W20260927 12:43:45.542116 392 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/krb5kdc/test-user.keytab.
Sep 27 12:43:45 dist-test-slave-bnc1 krb5kdc[15742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513025, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:43:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:43:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:43:45.665164 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43175
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43175
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:43:45.774567 15755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:43:45.774837 15755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:43:45.774878 15755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:43:45.778934 15755 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 12:43:45.778995 15755 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 12:43:45.779011 15755 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 12:43:45.779028 15755 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:43:45.779047 15755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:43:45.779084 15755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:43:45.779116 15755 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:43:45.779143 15755 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:43:45.784619 15755 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43175
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15755
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:43:45.785730 15755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:43:45.786794 15755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:43:45.795478 15763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:43:45.795506 15760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:43:45.795629 15761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:43:45.795653 15755 server_base.cc:1034] running on GCE node
I20260927 12:43:45.796186 15755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:43:45.797385 15755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:43:45.798568 15755 hybrid_clock.cc:648] HybridClock initialized: now 1790513025798542 us; error 51 us; skew 500 ppm
Sep 27 12:43:45 dist-test-slave-bnc1 krb5kdc[15742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513025, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:43:45.801276 15755 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:43:45.802119 15755 webserver.cc:460] Webserver started at http://127.0.98.62:40691/ using document root <none> and password file <none>
I20260927 12:43:45.802609 15755 fs_manager.cc:359] Metadata directory not provided
I20260927 12:43:45.802667 15755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:43:45.802817 15755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:43:45.804734 15755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/master-0/data/instance:
uuid: "6a84386bfb6447078525c886e771edd7"
format_stamp: "Formatted at 2026-09-27 12:43:45 on dist-test-slave-bnc1"
server_key: "fd8a4d08e16de8eba9ab390545f34ad1"
server_key_iv: "7d0076f3c5a54d20df6e644a4f337782"
server_key_version: "encryptionkey@0"
I20260927 12:43:45.805229 15755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/master-0/wal/instance:
uuid: "6a84386bfb6447078525c886e771edd7"
format_stamp: "Formatted at 2026-09-27 12:43:45 on dist-test-slave-bnc1"
server_key: "fd8a4d08e16de8eba9ab390545f34ad1"
server_key_iv: "7d0076f3c5a54d20df6e644a4f337782"
server_key_version: "encryptionkey@0"
I20260927 12:43:45.808856 15755 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 12:43:45.811376 15770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:43:45.812635 15755 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 12:43:45.812770 15755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/master-0/wal
uuid: "6a84386bfb6447078525c886e771edd7"
format_stamp: "Formatted at 2026-09-27 12:43:45 on dist-test-slave-bnc1"
server_key: "fd8a4d08e16de8eba9ab390545f34ad1"
server_key_iv: "7d0076f3c5a54d20df6e644a4f337782"
server_key_version: "encryptionkey@0"
I20260927 12:43:45.812876 15755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:43:45.827095 15755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:43:45.830329 15755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:43:45.830545 15755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:43:45.837198 15755 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43175
I20260927 12:43:45.837486 15822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43175 every 8 connection(s)
I20260927 12:43:45.838236 15755 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:43:45.841033 15823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:43:45.841195 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 15755
I20260927 12:43:45.841347 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/master-0/wal/instance
I20260927 12:43:45.841701 392 external_mini_cluster.cc:1428] Setting key d7a06722cb47c2c18381132f6fd960fb
I20260927 12:43:45.848169 15823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a84386bfb6447078525c886e771edd7: Bootstrap starting.
I20260927 12:43:45.850637 15823 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a84386bfb6447078525c886e771edd7: Neither blocks nor log segments found. Creating new log.
Sep 27 12:43:45 dist-test-slave-bnc1 krb5kdc[15742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513025, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:43:45.851405 15823 log.cc:824] T 00000000000000000000000000000000 P 6a84386bfb6447078525c886e771edd7: Log is configured to *not* fsync() on all Append() calls
I20260927 12:43:45.853770 15823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a84386bfb6447078525c886e771edd7: No bootstrap required, opened a new log
I20260927 12:43:45.856808 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:45.843072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45800 (local address 127.0.98.62:43175)
0927 12:43:45.843500 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0927 12:43:45.843519 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 12:43:45.843562 (+ 43us) server_negotiation.cc:407] Connection header received
0927 12:43:45.844390 (+ 828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:43:45.844414 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:43:45.844859 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:43:45.845233 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:43:45.845945 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:45.847449 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:45.848350 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:45.849019 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:45.849186 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:45.851678 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:43:45.851685 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:43:45.851701 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:43:45.851735 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:43:45.853732 (+ 1997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:45.854169 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:45.854175 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:45.854180 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:45.854321 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:45.854571 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:45.854574 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:45.854576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:45.855061 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:43:45.855254 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0927 12:43:45.855444 (+ 190us) server_negotiation.cc:299] Negotiation successful
0927 12:43:45.855736 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":126,"threads_started":1}
I20260927 12:43:45.858276 15823 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a84386bfb6447078525c886e771edd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a84386bfb6447078525c886e771edd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43175 } }
I20260927 12:43:45.858520 15823 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a84386bfb6447078525c886e771edd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:43:45.858583 15823 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a84386bfb6447078525c886e771edd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a84386bfb6447078525c886e771edd7, State: Initialized, Role: FOLLOWER
I20260927 12:43:45.859030 15823 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a84386bfb6447078525c886e771edd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a84386bfb6447078525c886e771edd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43175 } }
I20260927 12:43:45.859176 15823 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a84386bfb6447078525c886e771edd7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:43:45.859241 15823 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a84386bfb6447078525c886e771edd7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:43:45.859331 15823 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a84386bfb6447078525c886e771edd7 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:43:45.860209 15823 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a84386bfb6447078525c886e771edd7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a84386bfb6447078525c886e771edd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43175 } }
I20260927 12:43:45.860594 15823 leader_election.cc:302] T 00000000000000000000000000000000 P 6a84386bfb6447078525c886e771edd7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a84386bfb6447078525c886e771edd7; no voters:
I20260927 12:43:45.860936 15823 leader_election.cc:288] T 00000000000000000000000000000000 P 6a84386bfb6447078525c886e771edd7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:43:45.861027 15828 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a84386bfb6447078525c886e771edd7 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:43:45.861287 15828 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a84386bfb6447078525c886e771edd7 [term 1 LEADER]: Becoming Leader. State: Replica: 6a84386bfb6447078525c886e771edd7, State: Running, Role: LEADER
I20260927 12:43:45.861706 15828 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a84386bfb6447078525c886e771edd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a84386bfb6447078525c886e771edd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43175 } }
I20260927 12:43:45.862221 15823 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a84386bfb6447078525c886e771edd7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:43:45.863253 15830 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a84386bfb6447078525c886e771edd7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a84386bfb6447078525c886e771edd7. Latest consensus state: current_term: 1 leader_uuid: "6a84386bfb6447078525c886e771edd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a84386bfb6447078525c886e771edd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43175 } } }
I20260927 12:43:45.863397 15830 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a84386bfb6447078525c886e771edd7 [sys.catalog]: This master's current role is: LEADER
I20260927 12:43:45.863734 15829 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a84386bfb6447078525c886e771edd7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a84386bfb6447078525c886e771edd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a84386bfb6447078525c886e771edd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43175 } } }
I20260927 12:43:45.863821 15837 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:43:45.863890 15829 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a84386bfb6447078525c886e771edd7 [sys.catalog]: This master's current role is: LEADER
I20260927 12:43:45.866941 15837 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:43:45.872653 15837 catalog_manager.cc:1347] Generated new cluster ID: e5734dd828784514a0003d1ff75ee734
I20260927 12:43:45.872766 15837 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:43:45.889076 15837 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:43:45.890437 15837 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:43:45.897608 15837 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a84386bfb6447078525c886e771edd7: Generated new TSK 0
I20260927 12:43:45.898458 15837 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 12:43:45.937233 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43175
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:43:46.048527 15849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:43:46.048781 15849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:43:46.048823 15849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:43:46.052857 15849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:43:46.052913 15849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:43:46.053009 15849 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:43:46.058223 15849 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15849
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:43:46.059209 15849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:43:46.060231 15849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:43:46.069911 15855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:43:46.069904 15854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:43:46.069911 15857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:43:46.070209 15849 server_base.cc:1034] running on GCE node
I20260927 12:43:46.070653 15849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:43:46.071250 15849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:43:46.072449 15849 hybrid_clock.cc:648] HybridClock initialized: now 1790513026072431 us; error 36 us; skew 500 ppm
Sep 27 12:43:46 dist-test-slave-bnc1 krb5kdc[15742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513026, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:43:46.075258 15849 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 12:43:46.076139 15849 webserver.cc:460] Webserver started at http://127.0.98.1:41557/ using document root <none> and password file <none>
I20260927 12:43:46.076740 15849 fs_manager.cc:359] Metadata directory not provided
I20260927 12:43:46.076836 15849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:43:46.077083 15849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:43:46.079149 15849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-0/data/instance:
uuid: "313950c3398246619d5ca19a0372c094"
format_stamp: "Formatted at 2026-09-27 12:43:46 on dist-test-slave-bnc1"
server_key: "305ae662cc990c826d7cfb53b629b7eb"
server_key_iv: "1faf95d7b50307bb97c15d9fbe413e22"
server_key_version: "encryptionkey@0"
I20260927 12:43:46.079720 15849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-0/wal/instance:
uuid: "313950c3398246619d5ca19a0372c094"
format_stamp: "Formatted at 2026-09-27 12:43:46 on dist-test-slave-bnc1"
server_key: "305ae662cc990c826d7cfb53b629b7eb"
server_key_iv: "1faf95d7b50307bb97c15d9fbe413e22"
server_key_version: "encryptionkey@0"
I20260927 12:43:46.083698 15849 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 12:43:46.086179 15864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:43:46.087473 15849 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 12:43:46.087644 15849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-0/wal
uuid: "313950c3398246619d5ca19a0372c094"
format_stamp: "Formatted at 2026-09-27 12:43:46 on dist-test-slave-bnc1"
server_key: "305ae662cc990c826d7cfb53b629b7eb"
server_key_iv: "1faf95d7b50307bb97c15d9fbe413e22"
server_key_version: "encryptionkey@0"
I20260927 12:43:46.087769 15849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:43:46.116281 15849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:43:46.119840 15849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:43:46.120052 15849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:43:46.120663 15849 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:43:46.121503 15849 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:43:46.121606 15849 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:43:46.121703 15849 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:43:46.121757 15849 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:43:46.131309 15849 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34813
I20260927 12:43:46.131328 15977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34813 every 8 connection(s)
I20260927 12:43:46.132261 15849 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:43:46.133991 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 15849
I20260927 12:43:46.134123 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-0/wal/instance
I20260927 12:43:46.134465 392 external_mini_cluster.cc:1428] Setting key 1a70cc48e6b326a84756d1799c039dc1
Sep 27 12:43:46 dist-test-slave-bnc1 krb5kdc[15742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513026, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 12:43:46.147235 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:46.134180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34001 (local address 127.0.98.62:43175)
0927 12:43:46.134356 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 12:43:46.134362 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:43:46.135478 (+ 1116us) server_negotiation.cc:407] Connection header received
0927 12:43:46.136409 (+ 931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:43:46.136412 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:43:46.136462 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:43:46.136559 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:43:46.137847 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:46.138387 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:46.139604 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:46.140026 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:46.140163 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:46.142973 (+ 2810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:43:46.142977 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:43:46.142981 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:43:46.143014 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:43:46.144726 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:46.145346 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:46.145349 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:46.145350 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:46.145429 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:46.145928 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:46.145931 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:46.145933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:46.146137 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:43:46.146234 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 12:43:46.146900 (+ 666us) server_negotiation.cc:299] Negotiation successful
0927 12:43:46.147052 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 12:43:46.148082 15980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:46.134583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43175 (local address 127.0.98.1:34001)
0927 12:43:46.135315 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:46.135370 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 12:43:46.136198 (+ 828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:43:46.136755 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:43:46.136762 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:43:46.137146 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:43:46.137705 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:46.137721 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:46.138522 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:46.138525 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:46.139441 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:46.139450 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:46.140190 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:46.140193 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:46.140339 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:46.140987 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:43:46.141005 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:43:46.142784 (+ 1779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:43:46.144918 (+ 2134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:43:46.144928 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:43:46.144948 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:43:46.145252 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:43:46.145513 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:43:46.145516 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:43:46.145517 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:43:46.145712 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:43:46.146289 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:43:46.146299 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:43:46.146641 (+ 342us) client_negotiation.cc:769] Sending connection context
0927 12:43:46.146838 (+ 197us) client_negotiation.cc:240] Negotiation successful
0927 12:43:46.147202 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":136,"threads_started":1}
I20260927 12:43:46.149425 15978 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43175
I20260927 12:43:46.149725 15978 heartbeater.cc:464] Registering TS with master...
I20260927 12:43:46.150352 15978 heartbeater.cc:511] Master 127.0.98.62:43175 requested a full tablet report, sending...
I20260927 12:43:46.152297 15787 ts_manager.cc:194] Registered new tserver with Master: 313950c3398246619d5ca19a0372c094 (127.0.98.1:34813)
I20260927 12:43:46.154009 15787 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34001
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 12:43:46.164376 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43175
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:43:46.273873 15983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:43:46.274139 15983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:43:46.274190 15983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:43:46.278085 15983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:43:46.278133 15983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:43:46.278195 15983 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:43:46.283267 15983 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:43:46.284232 15983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:43:46.285269 15983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:43:46.294942 15988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:43:46.294909 15991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:43:46.294900 15989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:43:46.295127 15983 server_base.cc:1034] running on GCE node
I20260927 12:43:46.295575 15983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:43:46.296250 15983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:43:46.297458 15983 hybrid_clock.cc:648] HybridClock initialized: now 1790513026297442 us; error 31 us; skew 500 ppm
Sep 27 12:43:46 dist-test-slave-bnc1 krb5kdc[15742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513026, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:43:46.300508 15983 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 12:43:46.301424 15983 webserver.cc:460] Webserver started at http://127.0.98.2:38755/ using document root <none> and password file <none>
I20260927 12:43:46.302074 15983 fs_manager.cc:359] Metadata directory not provided
I20260927 12:43:46.302169 15983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:43:46.302407 15983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:43:46.304461 15983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-1/data/instance:
uuid: "1b5d8892eb234ddcb1e959d2c5617c2d"
format_stamp: "Formatted at 2026-09-27 12:43:46 on dist-test-slave-bnc1"
server_key: "be8c5a9d34a8afa35d67ab16958fd8c0"
server_key_iv: "dac9f3589347e525b205bf19d941c5a8"
server_key_version: "encryptionkey@0"
I20260927 12:43:46.305024 15983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1b5d8892eb234ddcb1e959d2c5617c2d"
format_stamp: "Formatted at 2026-09-27 12:43:46 on dist-test-slave-bnc1"
server_key: "be8c5a9d34a8afa35d67ab16958fd8c0"
server_key_iv: "dac9f3589347e525b205bf19d941c5a8"
server_key_version: "encryptionkey@0"
I20260927 12:43:46.308869 15983 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 12:43:46.311507 15998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:43:46.312753 15983 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 12:43:46.312887 15983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-1/wal
uuid: "1b5d8892eb234ddcb1e959d2c5617c2d"
format_stamp: "Formatted at 2026-09-27 12:43:46 on dist-test-slave-bnc1"
server_key: "be8c5a9d34a8afa35d67ab16958fd8c0"
server_key_iv: "dac9f3589347e525b205bf19d941c5a8"
server_key_version: "encryptionkey@0"
I20260927 12:43:46.313000 15983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:43:46.325389 15983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:43:46.328696 15983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:43:46.328900 15983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:43:46.329492 15983 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:43:46.330417 15983 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:43:46.330489 15983 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:43:46.330564 15983 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:43:46.330608 15983 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:43:46.339326 15983 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41007
I20260927 12:43:46.339393 16111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41007 every 8 connection(s)
I20260927 12:43:46.340284 15983 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:43:46.340468 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 15983
I20260927 12:43:46.340600 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-1/wal/instance
I20260927 12:43:46.340960 392 external_mini_cluster.cc:1428] Setting key 94a670b71e828589774d813cbfa5f2ea
Sep 27 12:43:46 dist-test-slave-bnc1 krb5kdc[15742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513026, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 12:43:46.355360 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:46.342203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41471 (local address 127.0.98.62:43175)
0927 12:43:46.342330 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 12:43:46.342336 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:43:46.343680 (+ 1344us) server_negotiation.cc:407] Connection header received
0927 12:43:46.344441 (+ 761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:43:46.344446 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:43:46.344508 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:43:46.344608 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:43:46.345820 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:46.346751 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:46.347920 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:46.348355 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:46.348476 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:46.351326 (+ 2850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:43:46.351329 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:43:46.351331 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:43:46.351356 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:43:46.352920 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:46.353450 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:46.353456 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:46.353459 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:46.353611 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:46.354050 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:46.354053 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:46.354055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:46.354321 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:43:46.354426 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 12:43:46.354933 (+ 507us) server_negotiation.cc:299] Negotiation successful
0927 12:43:46.355091 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 12:43:46.356330 16114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:46.342720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43175 (local address 127.0.98.2:41471)
0927 12:43:46.343412 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:46.343462 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 12:43:46.344246 (+ 784us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:43:46.344787 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:43:46.344793 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:43:46.345141 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:43:46.345672 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:46.345696 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:46.346875 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:46.346878 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:46.347786 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:46.347796 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:46.348505 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:46.348508 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:46.348682 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:46.349375 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:43:46.349392 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:43:46.351156 (+ 1764us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:43:46.353052 (+ 1896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:43:46.353063 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:43:46.353076 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:43:46.353341 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:43:46.353717 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:43:46.353723 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:43:46.353726 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:43:46.353925 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:43:46.354429 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:43:46.354434 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:43:46.354687 (+ 253us) client_negotiation.cc:769] Sending connection context
0927 12:43:46.354851 (+ 164us) client_negotiation.cc:240] Negotiation successful
0927 12:43:46.355137 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":135,"threads_started":1}
I20260927 12:43:46.357662 16112 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43175
I20260927 12:43:46.357959 16112 heartbeater.cc:464] Registering TS with master...
I20260927 12:43:46.358513 16112 heartbeater.cc:511] Master 127.0.98.62:43175 requested a full tablet report, sending...
I20260927 12:43:46.359781 15787 ts_manager.cc:194] Registered new tserver with Master: 1b5d8892eb234ddcb1e959d2c5617c2d (127.0.98.2:41007)
I20260927 12:43:46.360387 15787 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41471
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 12:43:46.370644 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43175
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:43:46.485296 16117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:43:46.485649 16117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:43:46.485706 16117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:43:46.489746 16117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:43:46.489822 16117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:43:46.489888 16117 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:43:46.495463 16117 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16117
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:43:46.496495 16117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:43:46.497660 16117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:43:46.507484 16125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:43:46.507493 16123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:43:46.507614 16117 server_base.cc:1034] running on GCE node
W20260927 12:43:46.507459 16122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:43:46.508025 16117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:43:46.508698 16117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:43:46.509891 16117 hybrid_clock.cc:648] HybridClock initialized: now 1790513026509875 us; error 27 us; skew 500 ppm
Sep 27 12:43:46 dist-test-slave-bnc1 krb5kdc[15742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513026, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:43:46.512806 16117 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 12:43:46.513721 16117 webserver.cc:460] Webserver started at http://127.0.98.3:45361/ using document root <none> and password file <none>
I20260927 12:43:46.514231 16117 fs_manager.cc:359] Metadata directory not provided
I20260927 12:43:46.514288 16117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:43:46.514441 16117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:43:46.516510 16117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-2/data/instance:
uuid: "9db0770e11084629879664a73186272b"
format_stamp: "Formatted at 2026-09-27 12:43:46 on dist-test-slave-bnc1"
server_key: "36222f23d19619549387b908d09b3d7c"
server_key_iv: "9cc54157df03191e36fcd45ff88b1bf2"
server_key_version: "encryptionkey@0"
I20260927 12:43:46.517150 16117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9db0770e11084629879664a73186272b"
format_stamp: "Formatted at 2026-09-27 12:43:46 on dist-test-slave-bnc1"
server_key: "36222f23d19619549387b908d09b3d7c"
server_key_iv: "9cc54157df03191e36fcd45ff88b1bf2"
server_key_version: "encryptionkey@0"
I20260927 12:43:46.520834 16117 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:43:46.523378 16132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:43:46.524670 16117 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 12:43:46.524812 16117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-2/wal
uuid: "9db0770e11084629879664a73186272b"
format_stamp: "Formatted at 2026-09-27 12:43:46 on dist-test-slave-bnc1"
server_key: "36222f23d19619549387b908d09b3d7c"
server_key_iv: "9cc54157df03191e36fcd45ff88b1bf2"
server_key_version: "encryptionkey@0"
I20260927 12:43:46.524930 16117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:43:46.550838 16117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:43:46.554263 16117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:43:46.554473 16117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:43:46.555078 16117 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:43:46.555902 16117 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:43:46.555971 16117 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:43:46.556041 16117 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:43:46.556089 16117 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:43:46.565023 16117 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41783
I20260927 12:43:46.565052 16245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41783 every 8 connection(s)
I20260927 12:43:46.565937 16117 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:43:46.567332 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 16117
I20260927 12:43:46.567464 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-2/wal/instance
I20260927 12:43:46.567817 392 external_mini_cluster.cc:1428] Setting key 1c080509fbbc337eb9ad9322fab11756
Sep 27 12:43:46 dist-test-slave-bnc1 krb5kdc[15742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513026, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:43:46.584225 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:46.569414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56779 (local address 127.0.98.62:43175)
0927 12:43:46.569534 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 12:43:46.569551 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 12:43:46.569606 (+ 55us) server_negotiation.cc:407] Connection header received
0927 12:43:46.570396 (+ 790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:43:46.570398 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:43:46.570442 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:43:46.570508 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:43:46.571642 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:46.572157 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:46.573231 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:46.573451 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:46.573532 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:46.580352 (+ 6820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:43:46.580356 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:43:46.580359 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:43:46.580385 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:43:46.582190 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:46.582707 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:46.582710 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:46.582712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:46.582799 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:46.583226 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:46.583229 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:46.583230 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:46.583435 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:43:46.583566 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 12:43:46.583995 (+ 429us) server_negotiation.cc:299] Negotiation successful
0927 12:43:46.584102 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 12:43:46.585103 16248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:46.568300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43175 (local address 127.0.98.3:56779)
0927 12:43:46.569141 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:46.569198 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 12:43:46.570215 (+ 1017us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:43:46.570693 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:43:46.570700 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:43:46.571043 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:43:46.571492 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:46.571505 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:46.572256 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:46.572259 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:46.573123 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:46.573131 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:46.573615 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:46.573619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:46.573797 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:46.578690 (+ 4893us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:43:46.578706 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:43:46.580177 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:43:46.582310 (+ 2133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:43:46.582316 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:43:46.582332 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:43:46.582592 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:43:46.582896 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:43:46.582899 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:43:46.582901 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:43:46.583095 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:43:46.583536 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:43:46.583541 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:43:46.583774 (+ 233us) client_negotiation.cc:769] Sending connection context
0927 12:43:46.583932 (+ 158us) client_negotiation.cc:240] Negotiation successful
0927 12:43:46.584285 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":155,"threads_started":1}
I20260927 12:43:46.586234 16246 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43175
I20260927 12:43:46.586476 16246 heartbeater.cc:464] Registering TS with master...
I20260927 12:43:46.586989 16246 heartbeater.cc:511] Master 127.0.98.62:43175 requested a full tablet report, sending...
I20260927 12:43:46.587985 15787 ts_manager.cc:194] Registered new tserver with Master: 9db0770e11084629879664a73186272b (127.0.98.3:41783)
I20260927 12:43:46.588541 15787 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56779
I20260927 12:43:46.593772 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 12:43:46.603046 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:46.595769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45812 (local address 127.0.98.62:43175)
0927 12:43:46.595889 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 12:43:46.595892 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:43:46.596093 (+ 201us) server_negotiation.cc:407] Connection header received
0927 12:43:46.596202 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:43:46.596205 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:43:46.596242 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:43:46.596308 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:43:46.597001 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:46.597507 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:46.598394 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:46.598596 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:46.598678 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:46.600154 (+ 1476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:43:46.600157 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:43:46.600159 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:43:46.600183 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:43:46.601442 (+ 1259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:46.601974 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:46.601976 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:46.601977 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:46.602054 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:43:46.602295 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:43:46.602297 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:43:46.602298 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:43:46.602505 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:43:46.602607 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 12:43:46.602839 (+ 232us) server_negotiation.cc:299] Negotiation successful
0927 12:43:46.602931 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 12:43:46.606308 15787 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45812:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 12:43:46.611116 15787 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:43:46.628552 16261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:46.623306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41007 (local address 127.0.0.1:47978)
0927 12:43:46.623749 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:46.623763 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:43:46.623856 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:43:46.624508 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:43:46.624512 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:43:46.624544 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:43:46.624686 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:46.624692 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:46.625889 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:46.625895 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:46.627056 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:46.627065 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:46.627951 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:46.627955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:46.628074 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:46.628115 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 12:43:46.628232 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 12:43:46.628368 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"mutex_wait_us":5,"spinlock_wait_cycles":896,"thread_start_us":82,"threads_started":1}
I20260927 12:43:46.628546 16262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:46.623735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47978 (local address 127.0.98.2:41007)
0927 12:43:46.624095 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0927 12:43:46.624106 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 12:43:46.624146 (+ 40us) server_negotiation.cc:407] Connection header received
0927 12:43:46.624212 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:43:46.624216 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:43:46.624362 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:43:46.624836 (+ 474us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:43:46.624997 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:46.625765 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:46.627187 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:46.627821 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:46.627933 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:46.628087 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 12:43:46.628289 (+ 202us) server_negotiation.cc:299] Negotiation successful
0927 12:43:46.628392 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":160,"threads_started":1}
I20260927 12:43:46.629223 16257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:46.622523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41783 (local address 127.0.0.1:60388)
0927 12:43:46.623779 (+ 1256us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:46.623792 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:43:46.623877 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:43:46.624478 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:43:46.624483 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:43:46.624506 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:43:46.624652 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:46.624660 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:46.626193 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:46.626196 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:46.627563 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:46.627574 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:46.628827 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:46.628830 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:46.628918 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:46.628939 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 12:43:46.629014 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 12:43:46.629089 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"mutex_wait_us":70,"thread_start_us":93,"threads_started":1}
I20260927 12:43:46.629626 16258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:46.622689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60388 (local address 127.0.98.3:41783)
0927 12:43:46.623054 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0927 12:43:46.623066 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 12:43:46.623976 (+ 910us) server_negotiation.cc:407] Connection header received
0927 12:43:46.624065 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:43:46.624070 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:43:46.624292 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:43:46.624467 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:43:46.624783 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:46.626044 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:46.627707 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:46.628473 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:46.628596 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:46.628769 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 12:43:46.629063 (+ 294us) server_negotiation.cc:299] Negotiation successful
0927 12:43:46.629463 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":141,"threads_started":1}
I20260927 12:43:46.629840 16260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:46.623239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53978 (local address 127.0.98.1:34813)
0927 12:43:46.624093 (+ 854us) server_negotiation.cc:206] Beginning negotiation
0927 12:43:46.624107 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 12:43:46.624137 (+ 30us) server_negotiation.cc:407] Connection header received
0927 12:43:46.624198 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:43:46.624202 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:43:46.624348 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:43:46.624516 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:43:46.624882 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:46.626024 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:46.627698 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:46.628672 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:46.628778 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:46.628905 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 12:43:46.629511 (+ 606us) server_negotiation.cc:299] Negotiation successful
0927 12:43:46.629631 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":748,"thread_start_us":89,"threads_started":1}
I20260927 12:43:46.630097 16259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:46.622919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34813 (local address 127.0.0.1:53978)
0927 12:43:46.623398 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:46.623451 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 12:43:46.623571 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:43:46.624676 (+ 1105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:43:46.624679 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:43:46.624693 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:43:46.624785 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:46.624791 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:46.626193 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:46.626196 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:46.627581 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:46.627587 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:46.628959 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:46.628962 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:46.629068 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:46.629094 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 12:43:46.629315 (+ 221us) client_negotiation.cc:240] Negotiation successful
0927 12:43:46.629403 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"mutex_wait_us":213,"thread_start_us":91,"threads_started":1}
I20260927 12:43:46.632347 16046 tablet_service.cc:1467] Processing CreateTablet for tablet 90021b2cf8fa4612b86c298ed9d57292 (DEFAULT_TABLE table=test-table [id=99cc6a90c628437b82c989f6c9a7e742]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:43:46.632615 16180 tablet_service.cc:1467] Processing CreateTablet for tablet 90021b2cf8fa4612b86c298ed9d57292 (DEFAULT_TABLE table=test-table [id=99cc6a90c628437b82c989f6c9a7e742]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:43:46.632824 15912 tablet_service.cc:1467] Processing CreateTablet for tablet 90021b2cf8fa4612b86c298ed9d57292 (DEFAULT_TABLE table=test-table [id=99cc6a90c628437b82c989f6c9a7e742]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:43:46.633514 16180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90021b2cf8fa4612b86c298ed9d57292. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:43:46.633713 16046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90021b2cf8fa4612b86c298ed9d57292. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:43:46.633733 15912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90021b2cf8fa4612b86c298ed9d57292. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:43:46.639575 16263 tablet_bootstrap.cc:492] T 90021b2cf8fa4612b86c298ed9d57292 P 9db0770e11084629879664a73186272b: Bootstrap starting.
I20260927 12:43:46.641317 16265 tablet_bootstrap.cc:492] T 90021b2cf8fa4612b86c298ed9d57292 P 313950c3398246619d5ca19a0372c094: Bootstrap starting.
I20260927 12:43:46.642019 16263 tablet_bootstrap.cc:654] T 90021b2cf8fa4612b86c298ed9d57292 P 9db0770e11084629879664a73186272b: Neither blocks nor log segments found. Creating new log.
I20260927 12:43:46.643421 16263 log.cc:824] T 90021b2cf8fa4612b86c298ed9d57292 P 9db0770e11084629879664a73186272b: Log is configured to *not* fsync() on all Append() calls
I20260927 12:43:46.643657 16265 tablet_bootstrap.cc:654] T 90021b2cf8fa4612b86c298ed9d57292 P 313950c3398246619d5ca19a0372c094: Neither blocks nor log segments found. Creating new log.
I20260927 12:43:46.644084 16264 tablet_bootstrap.cc:492] T 90021b2cf8fa4612b86c298ed9d57292 P 1b5d8892eb234ddcb1e959d2c5617c2d: Bootstrap starting.
I20260927 12:43:46.644840 16265 log.cc:824] T 90021b2cf8fa4612b86c298ed9d57292 P 313950c3398246619d5ca19a0372c094: Log is configured to *not* fsync() on all Append() calls
I20260927 12:43:46.645499 16263 tablet_bootstrap.cc:492] T 90021b2cf8fa4612b86c298ed9d57292 P 9db0770e11084629879664a73186272b: No bootstrap required, opened a new log
I20260927 12:43:46.645735 16263 ts_tablet_manager.cc:1397] T 90021b2cf8fa4612b86c298ed9d57292 P 9db0770e11084629879664a73186272b: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260927 12:43:46.646239 16264 tablet_bootstrap.cc:654] T 90021b2cf8fa4612b86c298ed9d57292 P 1b5d8892eb234ddcb1e959d2c5617c2d: Neither blocks nor log segments found. Creating new log.
I20260927 12:43:46.646855 16265 tablet_bootstrap.cc:492] T 90021b2cf8fa4612b86c298ed9d57292 P 313950c3398246619d5ca19a0372c094: No bootstrap required, opened a new log
I20260927 12:43:46.646878 16264 log.cc:824] T 90021b2cf8fa4612b86c298ed9d57292 P 1b5d8892eb234ddcb1e959d2c5617c2d: Log is configured to *not* fsync() on all Append() calls
I20260927 12:43:46.647024 16265 ts_tablet_manager.cc:1397] T 90021b2cf8fa4612b86c298ed9d57292 P 313950c3398246619d5ca19a0372c094: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 12:43:46.648561 16264 tablet_bootstrap.cc:492] T 90021b2cf8fa4612b86c298ed9d57292 P 1b5d8892eb234ddcb1e959d2c5617c2d: No bootstrap required, opened a new log
I20260927 12:43:46.648792 16264 ts_tablet_manager.cc:1397] T 90021b2cf8fa4612b86c298ed9d57292 P 1b5d8892eb234ddcb1e959d2c5617c2d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 12:43:46.649784 16263 raft_consensus.cc:348] T 90021b2cf8fa4612b86c298ed9d57292 P 9db0770e11084629879664a73186272b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db0770e11084629879664a73186272b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41783 } } peers { permanent_uuid: "1b5d8892eb234ddcb1e959d2c5617c2d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41007 } } peers { permanent_uuid: "313950c3398246619d5ca19a0372c094" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34813 } }
I20260927 12:43:46.650063 16263 raft_consensus.cc:374] T 90021b2cf8fa4612b86c298ed9d57292 P 9db0770e11084629879664a73186272b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:43:46.650144 16263 raft_consensus.cc:729] T 90021b2cf8fa4612b86c298ed9d57292 P 9db0770e11084629879664a73186272b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9db0770e11084629879664a73186272b, State: Initialized, Role: FOLLOWER
I20260927 12:43:46.650607 16263 consensus_queue.cc:260] T 90021b2cf8fa4612b86c298ed9d57292 P 9db0770e11084629879664a73186272b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db0770e11084629879664a73186272b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41783 } } peers { permanent_uuid: "1b5d8892eb234ddcb1e959d2c5617c2d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41007 } } peers { permanent_uuid: "313950c3398246619d5ca19a0372c094" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34813 } }
I20260927 12:43:46.650893 16265 raft_consensus.cc:348] T 90021b2cf8fa4612b86c298ed9d57292 P 313950c3398246619d5ca19a0372c094 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db0770e11084629879664a73186272b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41783 } } peers { permanent_uuid: "1b5d8892eb234ddcb1e959d2c5617c2d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41007 } } peers { permanent_uuid: "313950c3398246619d5ca19a0372c094" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34813 } }
I20260927 12:43:46.651185 16265 raft_consensus.cc:374] T 90021b2cf8fa4612b86c298ed9d57292 P 313950c3398246619d5ca19a0372c094 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:43:46.651270 16265 raft_consensus.cc:729] T 90021b2cf8fa4612b86c298ed9d57292 P 313950c3398246619d5ca19a0372c094 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 313950c3398246619d5ca19a0372c094, State: Initialized, Role: FOLLOWER
I20260927 12:43:46.651297 16246 heartbeater.cc:503] Master 127.0.98.62:43175 was elected leader, sending a full tablet report...
I20260927 12:43:46.651614 16263 ts_tablet_manager.cc:1428] T 90021b2cf8fa4612b86c298ed9d57292 P 9db0770e11084629879664a73186272b: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 12:43:46.651765 16265 consensus_queue.cc:260] T 90021b2cf8fa4612b86c298ed9d57292 P 313950c3398246619d5ca19a0372c094 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db0770e11084629879664a73186272b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41783 } } peers { permanent_uuid: "1b5d8892eb234ddcb1e959d2c5617c2d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41007 } } peers { permanent_uuid: "313950c3398246619d5ca19a0372c094" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34813 } }
I20260927 12:43:46.652406 15978 heartbeater.cc:503] Master 127.0.98.62:43175 was elected leader, sending a full tablet report...
I20260927 12:43:46.652292 16264 raft_consensus.cc:348] T 90021b2cf8fa4612b86c298ed9d57292 P 1b5d8892eb234ddcb1e959d2c5617c2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db0770e11084629879664a73186272b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41783 } } peers { permanent_uuid: "1b5d8892eb234ddcb1e959d2c5617c2d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41007 } } peers { permanent_uuid: "313950c3398246619d5ca19a0372c094" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34813 } }
I20260927 12:43:46.652612 16264 raft_consensus.cc:374] T 90021b2cf8fa4612b86c298ed9d57292 P 1b5d8892eb234ddcb1e959d2c5617c2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:43:46.652714 16264 raft_consensus.cc:729] T 90021b2cf8fa4612b86c298ed9d57292 P 1b5d8892eb234ddcb1e959d2c5617c2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b5d8892eb234ddcb1e959d2c5617c2d, State: Initialized, Role: FOLLOWER
I20260927 12:43:46.652788 16265 ts_tablet_manager.cc:1428] T 90021b2cf8fa4612b86c298ed9d57292 P 313950c3398246619d5ca19a0372c094: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 12:43:46.653177 16264 consensus_queue.cc:260] T 90021b2cf8fa4612b86c298ed9d57292 P 1b5d8892eb234ddcb1e959d2c5617c2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db0770e11084629879664a73186272b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41783 } } peers { permanent_uuid: "1b5d8892eb234ddcb1e959d2c5617c2d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41007 } } peers { permanent_uuid: "313950c3398246619d5ca19a0372c094" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34813 } }
I20260927 12:43:46.653962 16112 heartbeater.cc:503] Master 127.0.98.62:43175 was elected leader, sending a full tablet report...
I20260927 12:43:46.654176 16264 ts_tablet_manager.cc:1428] T 90021b2cf8fa4612b86c298ed9d57292 P 1b5d8892eb234ddcb1e959d2c5617c2d: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260927 12:43:46.816815 16247 tablet.cc:2369] T 90021b2cf8fa4612b86c298ed9d57292 P 9db0770e11084629879664a73186272b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 12:43:46.841487 16113 tablet.cc:2369] T 90021b2cf8fa4612b86c298ed9d57292 P 1b5d8892eb234ddcb1e959d2c5617c2d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:43:46.850987 16271 raft_consensus.cc:482] T 90021b2cf8fa4612b86c298ed9d57292 P 1b5d8892eb234ddcb1e959d2c5617c2d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:43:46.851229 16271 raft_consensus.cc:504] T 90021b2cf8fa4612b86c298ed9d57292 P 1b5d8892eb234ddcb1e959d2c5617c2d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db0770e11084629879664a73186272b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41783 } } peers { permanent_uuid: "1b5d8892eb234ddcb1e959d2c5617c2d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41007 } } peers { permanent_uuid: "313950c3398246619d5ca19a0372c094" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34813 } }
I20260927 12:43:46.852428 16271 leader_election.cc:288] T 90021b2cf8fa4612b86c298ed9d57292 P 1b5d8892eb234ddcb1e959d2c5617c2d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9db0770e11084629879664a73186272b (127.0.98.3:41783), 313950c3398246619d5ca19a0372c094 (127.0.98.1:34813)
I20260927 12:43:46.856314 16114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:46.852706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41783 (local address 127.0.98.2:45049)
0927 12:43:46.852860 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:46.852877 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 12:43:46.852977 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:43:46.853303 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:43:46.853305 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:43:46.853326 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:43:46.853459 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:46.853469 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:46.854289 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:46.854292 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:46.855158 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:46.855167 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:46.855850 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:46.855854 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:46.855967 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:46.856014 (+ 47us) client_negotiation.cc:769] Sending connection context
0927 12:43:46.856092 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 12:43:46.856152 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65,"mutex_wait_us":12}
I20260927 12:43:46.856330 16258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:46.852814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45049 (local address 127.0.98.3:41783)
0927 12:43:46.852947 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 12:43:46.852953 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:43:46.852973 (+ 20us) server_negotiation.cc:407] Connection header received
0927 12:43:46.853120 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:43:46.853125 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:43:46.853195 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:43:46.853275 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:43:46.853596 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:46.854179 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:46.855281 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:46.855727 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:46.855808 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:46.855857 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 12:43:46.856123 (+ 266us) server_negotiation.cc:299] Negotiation successful
0927 12:43:46.856213 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 12:43:46.856428 16260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:46.852925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49663 (local address 127.0.98.1:34813)
0927 12:43:46.853056 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 12:43:46.853060 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:43:46.853219 (+ 159us) server_negotiation.cc:407] Connection header received
0927 12:43:46.853340 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:43:46.853343 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:43:46.853393 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:43:46.853476 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:43:46.853800 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:46.854337 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:46.855402 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:46.855837 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:46.855942 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:46.856002 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 12:43:46.856241 (+ 239us) server_negotiation.cc:299] Negotiation successful
0927 12:43:46.856308 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 12:43:46.856462 16272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:46.852742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34813 (local address 127.0.98.2:49663)
0927 12:43:46.853113 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:46.853126 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:43:46.853238 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:43:46.853499 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:43:46.853501 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:43:46.853537 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:43:46.853681 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:46.853688 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:46.854449 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:46.854451 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:46.855274 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:43:46.855283 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:43:46.855966 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:43:46.855968 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:43:46.856100 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:46.856124 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 12:43:46.856200 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 12:43:46.856260 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"mutex_wait_us":39,"thread_start_us":102,"threads_started":1}
I20260927 12:43:46.857014 16200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90021b2cf8fa4612b86c298ed9d57292" candidate_uuid: "1b5d8892eb234ddcb1e959d2c5617c2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9db0770e11084629879664a73186272b" is_pre_election: true
I20260927 12:43:46.857014 15932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90021b2cf8fa4612b86c298ed9d57292" candidate_uuid: "1b5d8892eb234ddcb1e959d2c5617c2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "313950c3398246619d5ca19a0372c094" is_pre_election: true
I20260927 12:43:46.857329 16200 raft_consensus.cc:2454] T 90021b2cf8fa4612b86c298ed9d57292 P 9db0770e11084629879664a73186272b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b5d8892eb234ddcb1e959d2c5617c2d in term 0.
I20260927 12:43:46.857329 15932 raft_consensus.cc:2454] T 90021b2cf8fa4612b86c298ed9d57292 P 313950c3398246619d5ca19a0372c094 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b5d8892eb234ddcb1e959d2c5617c2d in term 0.
I20260927 12:43:46.857882 16001 leader_election.cc:302] T 90021b2cf8fa4612b86c298ed9d57292 P 1b5d8892eb234ddcb1e959d2c5617c2d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b5d8892eb234ddcb1e959d2c5617c2d, 9db0770e11084629879664a73186272b; no voters:
I20260927 12:43:46.858171 16271 raft_consensus.cc:2781] T 90021b2cf8fa4612b86c298ed9d57292 P 1b5d8892eb234ddcb1e959d2c5617c2d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:43:46.858249 16271 raft_consensus.cc:482] T 90021b2cf8fa4612b86c298ed9d57292 P 1b5d8892eb234ddcb1e959d2c5617c2d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:43:46.858282 16271 raft_consensus.cc:3037] T 90021b2cf8fa4612b86c298ed9d57292 P 1b5d8892eb234ddcb1e959d2c5617c2d [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:43:46.859372 16271 raft_consensus.cc:504] T 90021b2cf8fa4612b86c298ed9d57292 P 1b5d8892eb234ddcb1e959d2c5617c2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db0770e11084629879664a73186272b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41783 } } peers { permanent_uuid: "1b5d8892eb234ddcb1e959d2c5617c2d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41007 } } peers { permanent_uuid: "313950c3398246619d5ca19a0372c094" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34813 } }
I20260927 12:43:46.859766 16271 leader_election.cc:288] T 90021b2cf8fa4612b86c298ed9d57292 P 1b5d8892eb234ddcb1e959d2c5617c2d [CANDIDATE]: Term 1 election: Requested vote from peers 9db0770e11084629879664a73186272b (127.0.98.3:41783), 313950c3398246619d5ca19a0372c094 (127.0.98.1:34813)
I20260927 12:43:46.860064 16200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90021b2cf8fa4612b86c298ed9d57292" candidate_uuid: "1b5d8892eb234ddcb1e959d2c5617c2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9db0770e11084629879664a73186272b"
I20260927 12:43:46.860229 16200 raft_consensus.cc:3037] T 90021b2cf8fa4612b86c298ed9d57292 P 9db0770e11084629879664a73186272b [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:43:46.860293 15932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90021b2cf8fa4612b86c298ed9d57292" candidate_uuid: "1b5d8892eb234ddcb1e959d2c5617c2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "313950c3398246619d5ca19a0372c094"
I20260927 12:43:46.860445 15932 raft_consensus.cc:3037] T 90021b2cf8fa4612b86c298ed9d57292 P 313950c3398246619d5ca19a0372c094 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:43:46.861344 16200 raft_consensus.cc:2454] T 90021b2cf8fa4612b86c298ed9d57292 P 9db0770e11084629879664a73186272b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b5d8892eb234ddcb1e959d2c5617c2d in term 1.
I20260927 12:43:46.861589 15932 raft_consensus.cc:2454] T 90021b2cf8fa4612b86c298ed9d57292 P 313950c3398246619d5ca19a0372c094 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b5d8892eb234ddcb1e959d2c5617c2d in term 1.
I20260927 12:43:46.861712 16001 leader_election.cc:302] T 90021b2cf8fa4612b86c298ed9d57292 P 1b5d8892eb234ddcb1e959d2c5617c2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b5d8892eb234ddcb1e959d2c5617c2d, 9db0770e11084629879664a73186272b; no voters:
I20260927 12:43:46.861889 16271 raft_consensus.cc:2781] T 90021b2cf8fa4612b86c298ed9d57292 P 1b5d8892eb234ddcb1e959d2c5617c2d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:43:46.862136 16271 raft_consensus.cc:686] T 90021b2cf8fa4612b86c298ed9d57292 P 1b5d8892eb234ddcb1e959d2c5617c2d [term 1 LEADER]: Becoming Leader. State: Replica: 1b5d8892eb234ddcb1e959d2c5617c2d, State: Running, Role: LEADER
I20260927 12:43:46.862474 16271 consensus_queue.cc:237] T 90021b2cf8fa4612b86c298ed9d57292 P 1b5d8892eb234ddcb1e959d2c5617c2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db0770e11084629879664a73186272b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41783 } } peers { permanent_uuid: "1b5d8892eb234ddcb1e959d2c5617c2d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41007 } } peers { permanent_uuid: "313950c3398246619d5ca19a0372c094" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34813 } }
I20260927 12:43:46.865662 15787 catalog_manager.cc:5495] T 90021b2cf8fa4612b86c298ed9d57292 P 1b5d8892eb234ddcb1e959d2c5617c2d reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b5d8892eb234ddcb1e959d2c5617c2d (127.0.98.2). New cstate: current_term: 1 leader_uuid: "1b5d8892eb234ddcb1e959d2c5617c2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db0770e11084629879664a73186272b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b5d8892eb234ddcb1e959d2c5617c2d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "313950c3398246619d5ca19a0372c094" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34813 } health_report { overall_health: UNKNOWN } } }
W20260927 12:43:46.883657 15979 tablet.cc:2369] T 90021b2cf8fa4612b86c298ed9d57292 P 313950c3398246619d5ca19a0372c094: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:43:46.925428 16262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:46.922000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47990 (local address 127.0.98.2:41007)
0927 12:43:46.922164 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 12:43:46.922168 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:43:46.922182 (+ 14us) server_negotiation.cc:407] Connection header received
0927 12:43:46.922264 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:43:46.922267 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:43:46.922324 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:43:46.922430 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:43:46.922675 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:46.923183 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:46.924016 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:43:46.924232 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:43:46.924319 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:43:46.924647 (+ 328us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:43:46.925081 (+ 434us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:43:46.925184 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 12:43:46.925236 (+ 52us) server_negotiation.cc:299] Negotiation successful
0927 12:43:46.925286 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 12:43:46.931393 15932 raft_consensus.cc:1261] T 90021b2cf8fa4612b86c298ed9d57292 P 313950c3398246619d5ca19a0372c094 [term 1 FOLLOWER]: Refusing update from remote peer 1b5d8892eb234ddcb1e959d2c5617c2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:43:46.931394 16200 raft_consensus.cc:1261] T 90021b2cf8fa4612b86c298ed9d57292 P 9db0770e11084629879664a73186272b [term 1 FOLLOWER]: Refusing update from remote peer 1b5d8892eb234ddcb1e959d2c5617c2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:43:46.932133 16273 consensus_queue.cc:1035] T 90021b2cf8fa4612b86c298ed9d57292 P 1b5d8892eb234ddcb1e959d2c5617c2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9db0770e11084629879664a73186272b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:43:46.932305 16271 consensus_queue.cc:1035] T 90021b2cf8fa4612b86c298ed9d57292 P 1b5d8892eb234ddcb1e959d2c5617c2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "313950c3398246619d5ca19a0372c094" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:43:46.940954 16276 mvcc.cc:205] Tried to move back new op lower bound from 7333941358303985664 to 7333941358031437824. Current Snapshot: MvccSnapshot[applied={T|T < 7333941358303985664}]
I20260927 12:43:46.949141 15785 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:43:46.949370 15785 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:43:46.952158 15785 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6a84386bfb6447078525c886e771edd7: Sending DeleteTablet for 3 replicas of tablet 90021b2cf8fa4612b86c298ed9d57292
I20260927 12:43:46.953109 16046 tablet_service.cc:1514] Processing DeleteTablet for tablet 90021b2cf8fa4612b86c298ed9d57292 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:43:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47978
I20260927 12:43:46.953109 16180 tablet_service.cc:1514] Processing DeleteTablet for tablet 90021b2cf8fa4612b86c298ed9d57292 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:43:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60388
I20260927 12:43:46.953341 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 15849
I20260927 12:43:46.954130 16285 tablet_replica.cc:330] T 90021b2cf8fa4612b86c298ed9d57292 P 9db0770e11084629879664a73186272b: stopping tablet replica
I20260927 12:43:46.954473 16285 raft_consensus.cc:2229] T 90021b2cf8fa4612b86c298ed9d57292 P 9db0770e11084629879664a73186272b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:43:46.954833 16285 raft_consensus.cc:2258] T 90021b2cf8fa4612b86c298ed9d57292 P 9db0770e11084629879664a73186272b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:43:46.955279 16286 tablet_replica.cc:330] T 90021b2cf8fa4612b86c298ed9d57292 P 1b5d8892eb234ddcb1e959d2c5617c2d: stopping tablet replica
I20260927 12:43:46.955580 16286 raft_consensus.cc:2229] T 90021b2cf8fa4612b86c298ed9d57292 P 1b5d8892eb234ddcb1e959d2c5617c2d [term 1 LEADER]: Raft consensus shutting down.
I20260927 12:43:46.955955 16286 raft_consensus.cc:2258] T 90021b2cf8fa4612b86c298ed9d57292 P 1b5d8892eb234ddcb1e959d2c5617c2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:43:46.956724 16285 ts_tablet_manager.cc:1905] T 90021b2cf8fa4612b86c298ed9d57292 P 9db0770e11084629879664a73186272b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:43:46.957680 16286 ts_tablet_manager.cc:1905] T 90021b2cf8fa4612b86c298ed9d57292 P 1b5d8892eb234ddcb1e959d2c5617c2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:43:46.960281 16285 ts_tablet_manager.cc:1918] T 90021b2cf8fa4612b86c298ed9d57292 P 9db0770e11084629879664a73186272b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:43:46.960397 16285 log.cc:1196] T 90021b2cf8fa4612b86c298ed9d57292 P 9db0770e11084629879664a73186272b: Deleting WAL directory at /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-2/wal/wals/90021b2cf8fa4612b86c298ed9d57292
I20260927 12:43:46.960743 16285 ts_tablet_manager.cc:1939] T 90021b2cf8fa4612b86c298ed9d57292 P 9db0770e11084629879664a73186272b: Deleting consensus metadata
I20260927 12:43:46.960983 16286 ts_tablet_manager.cc:1918] T 90021b2cf8fa4612b86c298ed9d57292 P 1b5d8892eb234ddcb1e959d2c5617c2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:43:46.961156 16286 log.cc:1196] T 90021b2cf8fa4612b86c298ed9d57292 P 1b5d8892eb234ddcb1e959d2c5617c2d: Deleting WAL directory at /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-1/wal/wals/90021b2cf8fa4612b86c298ed9d57292
I20260927 12:43:46.961639 16286 ts_tablet_manager.cc:1939] T 90021b2cf8fa4612b86c298ed9d57292 P 1b5d8892eb234ddcb1e959d2c5617c2d: Deleting consensus metadata
W20260927 12:43:46.962145 15771 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34813 (error 108)
I20260927 12:43:46.962239 15773 catalog_manager.cc:4841] TS 9db0770e11084629879664a73186272b (127.0.98.3:41783): tablet 90021b2cf8fa4612b86c298ed9d57292 (table test-table [id=99cc6a90c628437b82c989f6c9a7e742]) successfully deleted
I20260927 12:43:46.963222 16261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:43:46.962906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34813 (local address 127.0.0.1:53986)
0927 12:43:46.963036 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 12:43:46.963110 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 12:43:46.963374 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 15983
I20260927 12:43:46.963418 15773 catalog_manager.cc:4841] TS 1b5d8892eb234ddcb1e959d2c5617c2d (127.0.98.2:41007): tablet 90021b2cf8fa4612b86c298ed9d57292 (table test-table [id=99cc6a90c628437b82c989f6c9a7e742]) successfully deleted
W20260927 12:43:46.963521 15771 catalog_manager.cc:4584] TS 313950c3398246619d5ca19a0372c094 (127.0.98.1:34813): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 90021b2cf8fa4612b86c298ed9d57292: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34813: connect: Connection refused (error 111)
I20260927 12:43:46.969718 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 16117
I20260927 12:43:46.975335 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 15755
I20260927 12:44:06.982161 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43175
--webserver_interface=127.0.98.62
--webserver_port=40691
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43175
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:44:07.095065 16288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:44:07.095355 16288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:44:07.095397 16288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:44:07.099509 16288 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 12:44:07.099560 16288 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 12:44:07.099576 16288 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 12:44:07.099591 16288 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:44:07.099608 16288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:44:07.099627 16288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:44:07.099642 16288 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:44:07.099665 16288 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:44:07.104837 16288 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43175
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=40691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16288
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:44:07.105935 16288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:44:07.106938 16288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:44:07.119568 16295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:44:07.119701 16288 server_base.cc:1034] running on GCE node
W20260927 12:44:07.119575 16294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:44:07.119577 16297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:44:07.120484 16288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:44:07.121683 16288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:44:07.122862 16288 hybrid_clock.cc:648] HybridClock initialized: now 1790513047122848 us; error 36 us; skew 500 ppm
Sep 27 12:44:07 dist-test-slave-bnc1 krb5kdc[15742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513047, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:44:07.130625 16288 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:44:07.131479 16288 webserver.cc:460] Webserver started at http://127.0.98.62:40691/ using document root <none> and password file <none>
I20260927 12:44:07.132025 16288 fs_manager.cc:359] Metadata directory not provided
I20260927 12:44:07.132089 16288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:44:07.135838 16288 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.002s
I20260927 12:44:07.137910 16304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:44:07.139130 16288 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:44:07.139253 16288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/master-0/wal
uuid: "6a84386bfb6447078525c886e771edd7"
format_stamp: "Formatted at 2026-09-27 12:43:45 on dist-test-slave-bnc1"
server_key: "fd8a4d08e16de8eba9ab390545f34ad1"
server_key_iv: "7d0076f3c5a54d20df6e644a4f337782"
server_key_version: "encryptionkey@0"
I20260927 12:44:07.139879 16288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:44:07.159448 16288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:44:07.162295 16288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:44:07.162497 16288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:44:07.169420 16288 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43175
I20260927 12:44:07.169754 16356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43175 every 8 connection(s)
I20260927 12:44:07.170605 16288 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:44:07.174959 16357 sys_catalog.cc:263] Verifying existing consensus state
I20260927 12:44:07.176625 16357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a84386bfb6447078525c886e771edd7: Bootstrap starting.
I20260927 12:44:07.178961 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 16288
I20260927 12:44:07.179375 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:34813
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:43175
--webserver_port=41557
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 12:44:07.193701 16357 log.cc:824] T 00000000000000000000000000000000 P 6a84386bfb6447078525c886e771edd7: Log is configured to *not* fsync() on all Append() calls
I20260927 12:44:07.202467 16357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a84386bfb6447078525c886e771edd7: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 12:44:07.202886 16357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a84386bfb6447078525c886e771edd7: Bootstrap complete.
I20260927 12:44:07.207005 16357 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a84386bfb6447078525c886e771edd7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a84386bfb6447078525c886e771edd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43175 } }
I20260927 12:44:07.207427 16357 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a84386bfb6447078525c886e771edd7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a84386bfb6447078525c886e771edd7, State: Initialized, Role: FOLLOWER
I20260927 12:44:07.207962 16357 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a84386bfb6447078525c886e771edd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a84386bfb6447078525c886e771edd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43175 } }
I20260927 12:44:07.208115 16357 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a84386bfb6447078525c886e771edd7 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:44:07.208209 16357 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a84386bfb6447078525c886e771edd7 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:44:07.208328 16357 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a84386bfb6447078525c886e771edd7 [term 1 FOLLOWER]: Advancing to term 2
I20260927 12:44:07.210356 16357 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a84386bfb6447078525c886e771edd7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a84386bfb6447078525c886e771edd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43175 } }
I20260927 12:44:07.210706 16357 leader_election.cc:302] T 00000000000000000000000000000000 P 6a84386bfb6447078525c886e771edd7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a84386bfb6447078525c886e771edd7; no voters:
I20260927 12:44:07.211061 16357 leader_election.cc:288] T 00000000000000000000000000000000 P 6a84386bfb6447078525c886e771edd7 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 12:44:07.211148 16361 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a84386bfb6447078525c886e771edd7 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 12:44:07.211349 16361 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a84386bfb6447078525c886e771edd7 [term 2 LEADER]: Becoming Leader. State: Replica: 6a84386bfb6447078525c886e771edd7, State: Running, Role: LEADER
I20260927 12:44:07.211716 16361 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a84386bfb6447078525c886e771edd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a84386bfb6447078525c886e771edd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43175 } }
I20260927 12:44:07.212334 16357 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a84386bfb6447078525c886e771edd7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:44:07.213259 16362 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a84386bfb6447078525c886e771edd7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "6a84386bfb6447078525c886e771edd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a84386bfb6447078525c886e771edd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43175 } } }
I20260927 12:44:07.213404 16362 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a84386bfb6447078525c886e771edd7 [sys.catalog]: This master's current role is: LEADER
I20260927 12:44:07.213717 16363 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a84386bfb6447078525c886e771edd7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a84386bfb6447078525c886e771edd7. Latest consensus state: current_term: 2 leader_uuid: "6a84386bfb6447078525c886e771edd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a84386bfb6447078525c886e771edd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43175 } } }
I20260927 12:44:07.213832 16363 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a84386bfb6447078525c886e771edd7 [sys.catalog]: This master's current role is: LEADER
I20260927 12:44:07.214334 16370 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:44:07.219195 16370 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:44:07.219875 16370 catalog_manager.cc:1259] Loaded cluster ID: e5734dd828784514a0003d1ff75ee734
I20260927 12:44:07.219938 16370 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:44:07.222533 16370 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:44:07.223223 16370 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6a84386bfb6447078525c886e771edd7: Loaded TSK: 0
I20260927 12:44:07.242435 16370 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a84386bfb6447078525c886e771edd7: Generated new TSK 1
I20260927 12:44:07.242827 16370 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 12:44:07.295295 16359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:44:07.295606 16359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:44:07.295686 16359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:44:07.299789 16359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:44:07.299880 16359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:44:07.299970 16359 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:44:07.305450 16359 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:34813
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=41557
--tserver_master_addrs=127.0.98.62:43175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:44:07.306569 16359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:44:07.307593 16359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:44:07.317396 16384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:44:07.317548 16359 server_base.cc:1034] running on GCE node
W20260927 12:44:07.317368 16385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:44:07.317945 16387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:44:07.318322 16359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:44:07.318938 16359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:44:07.320122 16359 hybrid_clock.cc:648] HybridClock initialized: now 1790513047320096 us; error 44 us; skew 500 ppm
Sep 27 12:44:07 dist-test-slave-bnc1 krb5kdc[15742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513047, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:44:07.323024 16359 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 12:44:07.324031 16359 webserver.cc:460] Webserver started at http://127.0.98.1:41557/ using document root <none> and password file <none>
I20260927 12:44:07.324600 16359 fs_manager.cc:359] Metadata directory not provided
I20260927 12:44:07.324671 16359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:44:07.328649 16359 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.002s
I20260927 12:44:07.331082 16394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:44:07.332440 16359 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 12:44:07.332594 16359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-0/wal
uuid: "313950c3398246619d5ca19a0372c094"
format_stamp: "Formatted at 2026-09-27 12:43:46 on dist-test-slave-bnc1"
server_key: "305ae662cc990c826d7cfb53b629b7eb"
server_key_iv: "1faf95d7b50307bb97c15d9fbe413e22"
server_key_version: "encryptionkey@0"
I20260927 12:44:07.333176 16359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:44:07.355597 16359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:44:07.358879 16359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:44:07.359093 16359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:44:07.359680 16359 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:44:07.360823 16401 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 12:44:07.364850 16359 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 12:44:07.364908 16359 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260927 12:44:07.364953 16359 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 12:44:07.366707 16359 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 12:44:07.366765 16359 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260927 12:44:07.367004 16401 tablet_bootstrap.cc:492] T 90021b2cf8fa4612b86c298ed9d57292 P 313950c3398246619d5ca19a0372c094: Bootstrap starting.
I20260927 12:44:07.377584 16359 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34813
I20260927 12:44:07.377633 16508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34813 every 8 connection(s)
I20260927 12:44:07.378826 16359 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:44:07.386142 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 16359
I20260927 12:44:07.386637 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:41007
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:43175
--webserver_port=38755
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 12:44:07.387940 16401 log.cc:824] T 90021b2cf8fa4612b86c298ed9d57292 P 313950c3398246619d5ca19a0372c094: Log is configured to *not* fsync() on all Append() calls
Sep 27 12:44:07 dist-test-slave-bnc1 krb5kdc[15742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513047, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:44:07.399904 16512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:07.380979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43175 (local address 127.0.98.1:34633)
0927 12:44:07.381536 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:07.381630 (+ 94us) client_negotiation.cc:174] Beginning negotiation
0927 12:44:07.382471 (+ 841us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:44:07.383579 (+ 1108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:44:07.383587 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:44:07.384035 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:44:07.384507 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:07.384521 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:07.385930 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:07.385932 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:07.387023 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:07.387034 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:07.387784 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:07.387788 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:07.387985 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:07.388789 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:44:07.388815 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:44:07.390966 (+ 2151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:44:07.394316 (+ 3350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:44:07.394326 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:44:07.394342 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:44:07.394754 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:44:07.395496 (+ 742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:44:07.395500 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:44:07.395504 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:44:07.395697 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:44:07.396837 (+ 1140us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:44:07.396846 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:44:07.397194 (+ 348us) client_negotiation.cc:769] Sending connection context
0927 12:44:07.397434 (+ 240us) client_negotiation.cc:240] Negotiation successful
0927 12:44:07.398196 (+ 762us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":132,"threads_started":1}
I20260927 12:44:07.402307 16511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:07.380964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34633 (local address 127.0.98.62:43175)
0927 12:44:07.381354 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:07.381378 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:07.381762 (+ 384us) server_negotiation.cc:407] Connection header received
0927 12:44:07.382891 (+ 1129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:07.382903 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:07.383196 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:07.383536 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:44:07.384694 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:07.385757 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:07.387150 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:07.387633 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:07.387819 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:07.391256 (+ 3437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:44:07.391265 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:44:07.391284 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:44:07.391325 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:44:07.394156 (+ 2831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:44:07.395182 (+ 1026us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:44:07.395191 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:44:07.395199 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:44:07.395370 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:44:07.395822 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:44:07.395827 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:44:07.395830 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:44:07.396599 (+ 769us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:44:07.396754 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:07.400397 (+ 3643us) server_negotiation.cc:299] Negotiation successful
0927 12:44:07.401042 (+ 645us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":127,"threads_started":1}
I20260927 12:44:07.405918 16509 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43175
I20260927 12:44:07.406210 16509 heartbeater.cc:464] Registering TS with master...
I20260927 12:44:07.406916 16509 heartbeater.cc:511] Master 127.0.98.62:43175 requested a full tablet report, sending...
I20260927 12:44:07.410509 16321 ts_manager.cc:194] Registered new tserver with Master: 313950c3398246619d5ca19a0372c094 (127.0.98.1:34813)
I20260927 12:44:07.416118 16401 tablet_bootstrap.cc:492] T 90021b2cf8fa4612b86c298ed9d57292 P 313950c3398246619d5ca19a0372c094: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 12:44:07.416779 16401 tablet_bootstrap.cc:492] T 90021b2cf8fa4612b86c298ed9d57292 P 313950c3398246619d5ca19a0372c094: Bootstrap complete.
I20260927 12:44:07.416893 16321 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34633
I20260927 12:44:07.418463 16401 ts_tablet_manager.cc:1397] T 90021b2cf8fa4612b86c298ed9d57292 P 313950c3398246619d5ca19a0372c094: Time spent bootstrapping tablet: real 0.052s user 0.022s sys 0.011s
I20260927 12:44:07.422040 16401 raft_consensus.cc:348] T 90021b2cf8fa4612b86c298ed9d57292 P 313950c3398246619d5ca19a0372c094 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db0770e11084629879664a73186272b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41783 } } peers { permanent_uuid: "1b5d8892eb234ddcb1e959d2c5617c2d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41007 } } peers { permanent_uuid: "313950c3398246619d5ca19a0372c094" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34813 } }
I20260927 12:44:07.425135 16401 raft_consensus.cc:729] T 90021b2cf8fa4612b86c298ed9d57292 P 313950c3398246619d5ca19a0372c094 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 313950c3398246619d5ca19a0372c094, State: Initialized, Role: FOLLOWER
I20260927 12:44:07.425748 16401 consensus_queue.cc:260] T 90021b2cf8fa4612b86c298ed9d57292 P 313950c3398246619d5ca19a0372c094 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db0770e11084629879664a73186272b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41783 } } peers { permanent_uuid: "1b5d8892eb234ddcb1e959d2c5617c2d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41007 } } peers { permanent_uuid: "313950c3398246619d5ca19a0372c094" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34813 } }
I20260927 12:44:07.426451 16509 heartbeater.cc:503] Master 127.0.98.62:43175 was elected leader, sending a full tablet report...
I20260927 12:44:07.426833 16401 ts_tablet_manager.cc:1428] T 90021b2cf8fa4612b86c298ed9d57292 P 313950c3398246619d5ca19a0372c094: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260927 12:44:07.430181 16515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:07.414962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34813 (local address 127.0.0.1:42710)
0927 12:44:07.415342 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:07.415388 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 12:44:07.415521 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:44:07.425014 (+ 9493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:44:07.425020 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:44:07.425045 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:44:07.425192 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:07.425203 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:07.426661 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:07.426664 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:07.428159 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:07.428193 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:07.429515 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:07.429518 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:07.429677 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:07.429728 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 12:44:07.429847 (+ 119us) client_negotiation.cc:240] Negotiation successful
0927 12:44:07.430014 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":102,"threads_started":1}
I20260927 12:44:07.430904 16516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:07.415025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42710 (local address 127.0.98.1:34813)
0927 12:44:07.424432 (+ 9407us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:07.424448 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:07.424492 (+ 44us) server_negotiation.cc:407] Connection header received
0927 12:44:07.424577 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:07.424584 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:07.424809 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:07.424991 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:44:07.425370 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:07.426522 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:07.428338 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:07.429211 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:07.429315 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:07.429447 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:07.430633 (+ 1186us) server_negotiation.cc:299] Negotiation successful
0927 12:44:07.430747 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9280,"thread_start_us":107,"threads_started":1}
I20260927 12:44:07.432061 16443 tablet_service.cc:1514] Processing DeleteTablet for tablet 90021b2cf8fa4612b86c298ed9d57292 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:43:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42710
I20260927 12:44:07.433012 16518 tablet_replica.cc:330] stopping tablet replica
I20260927 12:44:07.433409 16518 raft_consensus.cc:2229] T 90021b2cf8fa4612b86c298ed9d57292 P 313950c3398246619d5ca19a0372c094 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:44:07.433754 16518 raft_consensus.cc:2258] T 90021b2cf8fa4612b86c298ed9d57292 P 313950c3398246619d5ca19a0372c094 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:44:07.434989 16518 ts_tablet_manager.cc:1905] T 90021b2cf8fa4612b86c298ed9d57292 P 313950c3398246619d5ca19a0372c094: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:44:07.440554 16518 ts_tablet_manager.cc:1918] T 90021b2cf8fa4612b86c298ed9d57292 P 313950c3398246619d5ca19a0372c094: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:44:07.440693 16518 log.cc:1196] T 90021b2cf8fa4612b86c298ed9d57292 P 313950c3398246619d5ca19a0372c094: Deleting WAL directory at /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-0/wal/wals/90021b2cf8fa4612b86c298ed9d57292
I20260927 12:44:07.441059 16518 ts_tablet_manager.cc:1939] T 90021b2cf8fa4612b86c298ed9d57292 P 313950c3398246619d5ca19a0372c094: Deleting consensus metadata
I20260927 12:44:07.443360 16305 catalog_manager.cc:4841] TS 313950c3398246619d5ca19a0372c094 (127.0.98.1:34813): tablet 90021b2cf8fa4612b86c298ed9d57292 (table test-table [id=99cc6a90c628437b82c989f6c9a7e742]) successfully deleted
W20260927 12:44:07.539852 16513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:44:07.540154 16513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:44:07.540231 16513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:44:07.544528 16513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:44:07.544605 16513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:44:07.544691 16513 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:44:07.550398 16513 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:41007
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=38755
--tserver_master_addrs=127.0.98.62:43175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:44:07.551780 16513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:44:07.553058 16513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:44:07.562696 16524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:44:07.562753 16526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:44:07.562711 16523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:44:07.563082 16513 server_base.cc:1034] running on GCE node
I20260927 12:44:07.563441 16513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:44:07.564079 16513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:44:07.565237 16513 hybrid_clock.cc:648] HybridClock initialized: now 1790513047565203 us; error 48 us; skew 500 ppm
Sep 27 12:44:07 dist-test-slave-bnc1 krb5kdc[15742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513047, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:44:07.568069 16513 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 12:44:07.568926 16513 webserver.cc:460] Webserver started at http://127.0.98.2:38755/ using document root <none> and password file <none>
I20260927 12:44:07.569504 16513 fs_manager.cc:359] Metadata directory not provided
I20260927 12:44:07.569630 16513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:44:07.573242 16513 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:44:07.575241 16533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:44:07.576499 16513 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 12:44:07.576620 16513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-1/wal
uuid: "1b5d8892eb234ddcb1e959d2c5617c2d"
format_stamp: "Formatted at 2026-09-27 12:43:46 on dist-test-slave-bnc1"
server_key: "be8c5a9d34a8afa35d67ab16958fd8c0"
server_key_iv: "dac9f3589347e525b205bf19d941c5a8"
server_key_version: "encryptionkey@0"
I20260927 12:44:07.577157 16513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:44:07.608980 16513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:44:07.612522 16513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:44:07.612771 16513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:44:07.613499 16513 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:44:07.614459 16513 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:44:07.614534 16513 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:44:07.614599 16513 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:44:07.614634 16513 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:44:07.623636 16513 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41007
I20260927 12:44:07.623654 16646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41007 every 8 connection(s)
I20260927 12:44:07.624651 16513 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:44:07.629659 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 16513
I20260927 12:44:07.630178 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:41783
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:43175
--webserver_port=45361
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 12:44:07 dist-test-slave-bnc1 krb5kdc[15742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513047, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:44:07.638670 16511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:07.626449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53869 (local address 127.0.98.62:43175)
0927 12:44:07.626601 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:07.626605 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:07.627622 (+ 1017us) server_negotiation.cc:407] Connection header received
0927 12:44:07.628549 (+ 927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:07.628552 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:07.628613 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:07.628719 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:44:07.630171 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:07.630888 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:07.632252 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:07.632490 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:07.632607 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:07.635077 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:44:07.635081 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:44:07.635083 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:44:07.635109 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:44:07.636549 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:44:07.637065 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:44:07.637068 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:44:07.637070 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:44:07.637154 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:44:07.637514 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:44:07.637517 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:44:07.637521 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:44:07.637766 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:44:07.637868 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:07.638363 (+ 495us) server_negotiation.cc:299] Negotiation successful
0927 12:44:07.638495 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 12:44:07.639467 16649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:07.626843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43175 (local address 127.0.98.2:53869)
0927 12:44:07.627466 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:07.627505 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 12:44:07.628327 (+ 822us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:44:07.628895 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:44:07.628902 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:44:07.629259 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:44:07.629980 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:07.630000 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:07.631030 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:07.631037 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:07.632122 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:07.632132 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:07.632641 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:07.632645 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:07.632815 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:07.633411 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:44:07.633428 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:44:07.634896 (+ 1468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:44:07.636681 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:44:07.636686 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:44:07.636701 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:44:07.636957 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:44:07.637252 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:44:07.637255 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:44:07.637257 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:44:07.637434 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:44:07.637875 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:44:07.637883 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:44:07.638130 (+ 247us) client_negotiation.cc:769] Sending connection context
0927 12:44:07.638311 (+ 181us) client_negotiation.cc:240] Negotiation successful
0927 12:44:07.638642 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":104,"threads_started":1}
I20260927 12:44:07.640692 16647 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43175
I20260927 12:44:07.640945 16647 heartbeater.cc:464] Registering TS with master...
I20260927 12:44:07.641435 16647 heartbeater.cc:511] Master 127.0.98.62:43175 requested a full tablet report, sending...
I20260927 12:44:07.643425 16321 ts_manager.cc:194] Registered new tserver with Master: 1b5d8892eb234ddcb1e959d2c5617c2d (127.0.98.2:41007)
I20260927 12:44:07.644122 16321 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53869
W20260927 12:44:07.746031 16650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:44:07.746326 16650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:44:07.746404 16650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:44:07.750480 16650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:44:07.750587 16650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:44:07.750676 16650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:44:07.755975 16650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:41783
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=45361
--tserver_master_addrs=127.0.98.62:43175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16650
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:44:07.757004 16650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:44:07.758095 16650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:44:07.767716 16656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:44:07.767670 16655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:44:07.767670 16658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:44:07.767899 16650 server_base.cc:1034] running on GCE node
I20260927 12:44:07.768443 16650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:44:07.769088 16650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:44:07.770277 16650 hybrid_clock.cc:648] HybridClock initialized: now 1790513047770263 us; error 27 us; skew 500 ppm
Sep 27 12:44:07 dist-test-slave-bnc1 krb5kdc[15742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513047, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:44:07.772897 16650 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 12:44:07.773797 16650 webserver.cc:460] Webserver started at http://127.0.98.3:45361/ using document root <none> and password file <none>
I20260927 12:44:07.774340 16650 fs_manager.cc:359] Metadata directory not provided
I20260927 12:44:07.774443 16650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:44:07.778196 16650 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 12:44:07.780215 16665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:44:07.781453 16650 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:44:07.781618 16650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-2/wal
uuid: "9db0770e11084629879664a73186272b"
format_stamp: "Formatted at 2026-09-27 12:43:46 on dist-test-slave-bnc1"
server_key: "36222f23d19619549387b908d09b3d7c"
server_key_iv: "9cc54157df03191e36fcd45ff88b1bf2"
server_key_version: "encryptionkey@0"
I20260927 12:44:07.782167 16650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:44:07.806056 16650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:44:07.809412 16650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:44:07.809644 16650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:44:07.810249 16650 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:44:07.811066 16650 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:44:07.811132 16650 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:44:07.811203 16650 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:44:07.811252 16650 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:44:07.819936 16650 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41783
I20260927 12:44:07.819963 16778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41783 every 8 connection(s)
I20260927 12:44:07.820909 16650 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:44:07.826656 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 16650
Sep 27 12:44:07 dist-test-slave-bnc1 krb5kdc[15742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513047, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:44:07.834764 16511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:07.822784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44025 (local address 127.0.98.62:43175)
0927 12:44:07.822934 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:07.822938 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:07.824097 (+ 1159us) server_negotiation.cc:407] Connection header received
0927 12:44:07.824932 (+ 835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:07.824935 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:07.824983 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:07.825075 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:44:07.826338 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:07.827108 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:07.828406 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:07.828641 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:07.828754 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:07.831239 (+ 2485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:44:07.831243 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:44:07.831246 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:44:07.831271 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:44:07.832650 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:44:07.833175 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:44:07.833180 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:44:07.833181 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:44:07.833262 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:44:07.833630 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:44:07.833635 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:44:07.833636 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:44:07.833843 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:44:07.833948 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:07.834493 (+ 545us) server_negotiation.cc:299] Negotiation successful
0927 12:44:07.834631 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 12:44:07.835655 16782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:07.827319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37842 (local address 127.0.98.62:43175)
0927 12:44:07.827564 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:07.827568 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:07.827738 (+ 170us) server_negotiation.cc:407] Connection header received
0927 12:44:07.827886 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:07.827890 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:07.827940 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:07.828049 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:44:07.828777 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:07.829284 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:07.830196 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:07.830437 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:07.830544 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:07.832302 (+ 1758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:44:07.832304 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:44:07.832306 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:44:07.832327 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:44:07.833754 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:44:07.834193 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:44:07.834196 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:44:07.834198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:44:07.834282 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:44:07.834586 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:44:07.834590 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:44:07.834591 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:44:07.834791 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:44:07.834977 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:07.835398 (+ 421us) server_negotiation.cc:299] Negotiation successful
0927 12:44:07.835507 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":97,"threads_started":1}
I20260927 12:44:07.835688 16781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:07.823238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43175 (local address 127.0.98.3:44025)
0927 12:44:07.823895 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:07.823964 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0927 12:44:07.824750 (+ 786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:44:07.825238 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:44:07.825245 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:44:07.825737 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:44:07.826210 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:07.826222 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:07.827270 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:07.827274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:07.828274 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:07.828283 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:07.828792 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:07.828796 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:07.828960 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:07.829544 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:44:07.829560 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:44:07.831049 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:44:07.832773 (+ 1724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:44:07.832779 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:44:07.832794 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:44:07.833068 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:44:07.833356 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:44:07.833360 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:44:07.833362 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:44:07.833506 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:44:07.833937 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:44:07.833942 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:44:07.834213 (+ 271us) client_negotiation.cc:769] Sending connection context
0927 12:44:07.834414 (+ 201us) client_negotiation.cc:240] Negotiation successful
0927 12:44:07.834736 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":112,"threads_started":1}
I20260927 12:44:07.837064 16779 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43175
I20260927 12:44:07.837314 16779 heartbeater.cc:464] Registering TS with master...
I20260927 12:44:07.837944 16779 heartbeater.cc:511] Master 127.0.98.62:43175 requested a full tablet report, sending...
I20260927 12:44:07.838936 16321 ts_manager.cc:194] Registered new tserver with Master: 9db0770e11084629879664a73186272b (127.0.98.3:41783)
I20260927 12:44:07.839490 16321 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44025
I20260927 12:44:07.849290 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 12:44:07.853077 16782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:07.849948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37858 (local address 127.0.98.62:43175)
0927 12:44:07.850084 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:07.850090 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:07.850250 (+ 160us) server_negotiation.cc:407] Connection header received
0927 12:44:07.850386 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:07.850388 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:07.850483 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:07.850569 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:44:07.850791 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:07.851293 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:07.852075 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:07.852300 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:07.852372 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:07.852612 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:44:07.852742 (+ 130us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0927 12:44:07.852936 (+ 194us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37858: token expired
Metrics: {"server-negotiator.queue_time_us":51}
W20260927 12:44:07.853158 16782 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37858: token expired
W20260927 12:44:07.853447 16250 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:43175)
I20260927 12:44:07.860730 16782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:07.853867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37874 (local address 127.0.98.62:43175)
0927 12:44:07.853958 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:07.853962 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:07.853976 (+ 14us) server_negotiation.cc:407] Connection header received
0927 12:44:07.854049 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:07.854051 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:07.854092 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:07.854152 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:44:07.854718 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:07.855180 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:07.855917 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:07.856118 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:07.856187 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:07.857426 (+ 1239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:44:07.857429 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:44:07.857432 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:44:07.857454 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:44:07.858993 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:44:07.859463 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:44:07.859468 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:44:07.859470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:44:07.859557 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:44:07.859866 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:44:07.859869 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:44:07.859870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:44:07.860103 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:44:07.860209 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:07.860501 (+ 292us) server_negotiation.cc:299] Negotiation successful
0927 12:44:07.860605 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260927 12:44:07.864019 16321 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37874:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 12:44:07.869268 16321 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:44:07.878103 16443 tablet_service.cc:1467] Processing CreateTablet for tablet a469cf9350c14bdcbdf012fed32f42e3 (DEFAULT_TABLE table=test-table [id=9345e991a9de42e9985923e1ff3141c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:44:07.878756 16443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a469cf9350c14bdcbdf012fed32f42e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:44:07.882125 16401 tablet_bootstrap.cc:492] T a469cf9350c14bdcbdf012fed32f42e3 P 313950c3398246619d5ca19a0372c094: Bootstrap starting.
I20260927 12:44:07.882905 16515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:07.877155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41783 (local address 127.0.0.1:50968)
0927 12:44:07.877724 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:07.877740 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 12:44:07.877815 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:44:07.878977 (+ 1162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:44:07.878979 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:44:07.878996 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:44:07.879113 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:07.879120 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:07.880313 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:07.880319 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:07.881260 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:07.881287 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:07.882505 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:07.882509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:07.882599 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:07.882622 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 12:44:07.882688 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 12:44:07.882773 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497}
I20260927 12:44:07.882952 16788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:07.877340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50968 (local address 127.0.98.3:41783)
0927 12:44:07.878189 (+ 849us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:07.878202 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:07.878226 (+ 24us) server_negotiation.cc:407] Connection header received
0927 12:44:07.878358 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:07.878364 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:07.878751 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:07.878991 (+ 240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:44:07.879246 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:07.880205 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:07.881432 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:07.882374 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:07.882497 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:07.882656 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:07.882726 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 12:44:07.882816 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":744,"thread_start_us":141,"threads_started":1}
I20260927 12:44:07.883021 16401 tablet_bootstrap.cc:654] T a469cf9350c14bdcbdf012fed32f42e3 P 313950c3398246619d5ca19a0372c094: Neither blocks nor log segments found. Creating new log.
I20260927 12:44:07.884279 16401 tablet_bootstrap.cc:492] T a469cf9350c14bdcbdf012fed32f42e3 P 313950c3398246619d5ca19a0372c094: No bootstrap required, opened a new log
I20260927 12:44:07.884375 16401 ts_tablet_manager.cc:1397] T a469cf9350c14bdcbdf012fed32f42e3 P 313950c3398246619d5ca19a0372c094: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260927 12:44:07.885136 16401 raft_consensus.cc:348] T a469cf9350c14bdcbdf012fed32f42e3 P 313950c3398246619d5ca19a0372c094 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db0770e11084629879664a73186272b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41783 } } peers { permanent_uuid: "1b5d8892eb234ddcb1e959d2c5617c2d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41007 } } peers { permanent_uuid: "313950c3398246619d5ca19a0372c094" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34813 } }
I20260927 12:44:07.885265 16401 raft_consensus.cc:374] T a469cf9350c14bdcbdf012fed32f42e3 P 313950c3398246619d5ca19a0372c094 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:44:07.885310 16401 raft_consensus.cc:729] T a469cf9350c14bdcbdf012fed32f42e3 P 313950c3398246619d5ca19a0372c094 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 313950c3398246619d5ca19a0372c094, State: Initialized, Role: FOLLOWER
I20260927 12:44:07.885437 16401 consensus_queue.cc:260] T a469cf9350c14bdcbdf012fed32f42e3 P 313950c3398246619d5ca19a0372c094 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db0770e11084629879664a73186272b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41783 } } peers { permanent_uuid: "1b5d8892eb234ddcb1e959d2c5617c2d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41007 } } peers { permanent_uuid: "313950c3398246619d5ca19a0372c094" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34813 } }
I20260927 12:44:07.885751 16401 ts_tablet_manager.cc:1428] T a469cf9350c14bdcbdf012fed32f42e3 P 313950c3398246619d5ca19a0372c094: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260927 12:44:07.886598 16713 tablet_service.cc:1467] Processing CreateTablet for tablet a469cf9350c14bdcbdf012fed32f42e3 (DEFAULT_TABLE table=test-table [id=9345e991a9de42e9985923e1ff3141c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:44:07.887676 16713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a469cf9350c14bdcbdf012fed32f42e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:44:07.888161 16789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:07.877411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41007 (local address 127.0.0.1:49966)
0927 12:44:07.878700 (+ 1289us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:07.878716 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 12:44:07.878801 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:44:07.883302 (+ 4501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:44:07.883306 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:44:07.883324 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:44:07.883426 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:07.883431 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:07.884613 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:07.884617 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:07.886199 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:07.886211 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:07.887688 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:07.887698 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:07.887811 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:07.887836 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 12:44:07.887933 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 12:44:07.888022 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1216,"thread_start_us":114,"threads_started":1}
I20260927 12:44:07.888188 16790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:07.878147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49966 (local address 127.0.98.2:41007)
0927 12:44:07.882780 (+ 4633us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:07.882792 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:07.882817 (+ 25us) server_negotiation.cc:407] Connection header received
0927 12:44:07.882898 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:07.882906 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:07.883151 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:07.883317 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:44:07.883506 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:07.884492 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:07.886338 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:07.887345 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:07.887454 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:07.887626 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:07.887969 (+ 343us) server_negotiation.cc:299] Negotiation successful
0927 12:44:07.888063 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4523,"thread_start_us":112,"threads_started":1}
I20260927 12:44:07.891147 16581 tablet_service.cc:1467] Processing CreateTablet for tablet a469cf9350c14bdcbdf012fed32f42e3 (DEFAULT_TABLE table=test-table [id=9345e991a9de42e9985923e1ff3141c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:44:07.892042 16581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a469cf9350c14bdcbdf012fed32f42e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:44:07.894941 16792 tablet_bootstrap.cc:492] T a469cf9350c14bdcbdf012fed32f42e3 P 9db0770e11084629879664a73186272b: Bootstrap starting.
I20260927 12:44:07.896986 16792 tablet_bootstrap.cc:654] T a469cf9350c14bdcbdf012fed32f42e3 P 9db0770e11084629879664a73186272b: Neither blocks nor log segments found. Creating new log.
I20260927 12:44:07.898130 16792 log.cc:824] T a469cf9350c14bdcbdf012fed32f42e3 P 9db0770e11084629879664a73186272b: Log is configured to *not* fsync() on all Append() calls
I20260927 12:44:07.898903 16794 tablet_bootstrap.cc:492] T a469cf9350c14bdcbdf012fed32f42e3 P 1b5d8892eb234ddcb1e959d2c5617c2d: Bootstrap starting.
I20260927 12:44:07.900009 16792 tablet_bootstrap.cc:492] T a469cf9350c14bdcbdf012fed32f42e3 P 9db0770e11084629879664a73186272b: No bootstrap required, opened a new log
I20260927 12:44:07.900218 16792 ts_tablet_manager.cc:1397] T a469cf9350c14bdcbdf012fed32f42e3 P 9db0770e11084629879664a73186272b: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260927 12:44:07.902022 16794 tablet_bootstrap.cc:654] T a469cf9350c14bdcbdf012fed32f42e3 P 1b5d8892eb234ddcb1e959d2c5617c2d: Neither blocks nor log segments found. Creating new log.
I20260927 12:44:07.902741 16794 log.cc:824] T a469cf9350c14bdcbdf012fed32f42e3 P 1b5d8892eb234ddcb1e959d2c5617c2d: Log is configured to *not* fsync() on all Append() calls
I20260927 12:44:07.903424 16792 raft_consensus.cc:348] T a469cf9350c14bdcbdf012fed32f42e3 P 9db0770e11084629879664a73186272b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db0770e11084629879664a73186272b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41783 } } peers { permanent_uuid: "1b5d8892eb234ddcb1e959d2c5617c2d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41007 } } peers { permanent_uuid: "313950c3398246619d5ca19a0372c094" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34813 } }
I20260927 12:44:07.903734 16792 raft_consensus.cc:374] T a469cf9350c14bdcbdf012fed32f42e3 P 9db0770e11084629879664a73186272b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:44:07.903828 16792 raft_consensus.cc:729] T a469cf9350c14bdcbdf012fed32f42e3 P 9db0770e11084629879664a73186272b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9db0770e11084629879664a73186272b, State: Initialized, Role: FOLLOWER
I20260927 12:44:07.904255 16792 consensus_queue.cc:260] T a469cf9350c14bdcbdf012fed32f42e3 P 9db0770e11084629879664a73186272b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db0770e11084629879664a73186272b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41783 } } peers { permanent_uuid: "1b5d8892eb234ddcb1e959d2c5617c2d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41007 } } peers { permanent_uuid: "313950c3398246619d5ca19a0372c094" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34813 } }
I20260927 12:44:07.904917 16779 heartbeater.cc:503] Master 127.0.98.62:43175 was elected leader, sending a full tablet report...
I20260927 12:44:07.905087 16794 tablet_bootstrap.cc:492] T a469cf9350c14bdcbdf012fed32f42e3 P 1b5d8892eb234ddcb1e959d2c5617c2d: No bootstrap required, opened a new log
I20260927 12:44:07.905273 16792 ts_tablet_manager.cc:1428] T a469cf9350c14bdcbdf012fed32f42e3 P 9db0770e11084629879664a73186272b: Time spent starting tablet: real 0.005s user 0.004s sys 0.002s
I20260927 12:44:07.905279 16794 ts_tablet_manager.cc:1397] T a469cf9350c14bdcbdf012fed32f42e3 P 1b5d8892eb234ddcb1e959d2c5617c2d: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20260927 12:44:07.908862 16794 raft_consensus.cc:348] T a469cf9350c14bdcbdf012fed32f42e3 P 1b5d8892eb234ddcb1e959d2c5617c2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db0770e11084629879664a73186272b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41783 } } peers { permanent_uuid: "1b5d8892eb234ddcb1e959d2c5617c2d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41007 } } peers { permanent_uuid: "313950c3398246619d5ca19a0372c094" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34813 } }
I20260927 12:44:07.909117 16794 raft_consensus.cc:374] T a469cf9350c14bdcbdf012fed32f42e3 P 1b5d8892eb234ddcb1e959d2c5617c2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:44:07.909161 16794 raft_consensus.cc:729] T a469cf9350c14bdcbdf012fed32f42e3 P 1b5d8892eb234ddcb1e959d2c5617c2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b5d8892eb234ddcb1e959d2c5617c2d, State: Initialized, Role: FOLLOWER
I20260927 12:44:07.909619 16794 consensus_queue.cc:260] T a469cf9350c14bdcbdf012fed32f42e3 P 1b5d8892eb234ddcb1e959d2c5617c2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db0770e11084629879664a73186272b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41783 } } peers { permanent_uuid: "1b5d8892eb234ddcb1e959d2c5617c2d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41007 } } peers { permanent_uuid: "313950c3398246619d5ca19a0372c094" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34813 } }
I20260927 12:44:07.910255 16647 heartbeater.cc:503] Master 127.0.98.62:43175 was elected leader, sending a full tablet report...
I20260927 12:44:07.910619 16794 ts_tablet_manager.cc:1428] T a469cf9350c14bdcbdf012fed32f42e3 P 1b5d8892eb234ddcb1e959d2c5617c2d: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260927 12:44:08.059161 16797 raft_consensus.cc:482] T a469cf9350c14bdcbdf012fed32f42e3 P 1b5d8892eb234ddcb1e959d2c5617c2d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:44:08.059422 16797 raft_consensus.cc:504] T a469cf9350c14bdcbdf012fed32f42e3 P 1b5d8892eb234ddcb1e959d2c5617c2d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db0770e11084629879664a73186272b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41783 } } peers { permanent_uuid: "1b5d8892eb234ddcb1e959d2c5617c2d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41007 } } peers { permanent_uuid: "313950c3398246619d5ca19a0372c094" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34813 } }
I20260927 12:44:08.060595 16797 leader_election.cc:288] T a469cf9350c14bdcbdf012fed32f42e3 P 1b5d8892eb234ddcb1e959d2c5617c2d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9db0770e11084629879664a73186272b (127.0.98.3:41783), 313950c3398246619d5ca19a0372c094 (127.0.98.1:34813)
I20260927 12:44:08.064271 16798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:08.060989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38145 (local address 127.0.98.1:34813)
0927 12:44:08.061296 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:08.061299 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:08.061310 (+ 11us) server_negotiation.cc:407] Connection header received
0927 12:44:08.061355 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:08.061357 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:08.061394 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:08.061492 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:44:08.061801 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:08.062312 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:08.063308 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:08.063715 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:08.063832 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:08.063900 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:08.064078 (+ 178us) server_negotiation.cc:299] Negotiation successful
0927 12:44:08.064149 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":103,"threads_started":1}
I20260927 12:44:08.064270 16649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:08.060875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34813 (local address 127.0.98.2:38145)
0927 12:44:08.061016 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:08.061034 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 12:44:08.061147 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:44:08.061505 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:44:08.061508 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:44:08.061536 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:44:08.061687 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:08.061693 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:08.062426 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:08.062431 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:08.063184 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:08.063192 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:08.063835 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:08.063839 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:08.063942 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:08.063968 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 12:44:08.064062 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 12:44:08.064133 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 12:44:08.064487 16799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:08.060875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41783 (local address 127.0.98.2:47153)
0927 12:44:08.061338 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:08.061350 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 12:44:08.061474 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:44:08.061787 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:44:08.061790 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:44:08.061805 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:44:08.061923 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:08.061928 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:08.062635 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:08.062640 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:08.063394 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:08.063401 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:08.064054 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:08.064058 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:08.064164 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:08.064192 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 12:44:08.064274 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 12:44:08.064342 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"mutex_wait_us":136,"thread_start_us":115,"threads_started":1}
I20260927 12:44:08.064478 16788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:08.061017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47153 (local address 127.0.98.3:41783)
0927 12:44:08.061157 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:08.061163 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:08.061442 (+ 279us) server_negotiation.cc:407] Connection header received
0927 12:44:08.061639 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:08.061643 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:08.061693 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:08.061778 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:44:08.062049 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:08.062519 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:08.063516 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:08.063931 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:08.064056 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:08.064117 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:08.064304 (+ 187us) server_negotiation.cc:299] Negotiation successful
0927 12:44:08.064374 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 12:44:08.065218 16733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a469cf9350c14bdcbdf012fed32f42e3" candidate_uuid: "1b5d8892eb234ddcb1e959d2c5617c2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9db0770e11084629879664a73186272b" is_pre_election: true
I20260927 12:44:08.065253 16463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a469cf9350c14bdcbdf012fed32f42e3" candidate_uuid: "1b5d8892eb234ddcb1e959d2c5617c2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "313950c3398246619d5ca19a0372c094" is_pre_election: true
I20260927 12:44:08.065600 16733 raft_consensus.cc:2454] T a469cf9350c14bdcbdf012fed32f42e3 P 9db0770e11084629879664a73186272b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b5d8892eb234ddcb1e959d2c5617c2d in term 0.
I20260927 12:44:08.065608 16463 raft_consensus.cc:2454] T a469cf9350c14bdcbdf012fed32f42e3 P 313950c3398246619d5ca19a0372c094 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b5d8892eb234ddcb1e959d2c5617c2d in term 0.
I20260927 12:44:08.066124 16536 leader_election.cc:302] T a469cf9350c14bdcbdf012fed32f42e3 P 1b5d8892eb234ddcb1e959d2c5617c2d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1b5d8892eb234ddcb1e959d2c5617c2d, 313950c3398246619d5ca19a0372c094, 9db0770e11084629879664a73186272b; no voters:
I20260927 12:44:08.066413 16797 raft_consensus.cc:2781] T a469cf9350c14bdcbdf012fed32f42e3 P 1b5d8892eb234ddcb1e959d2c5617c2d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:44:08.066493 16797 raft_consensus.cc:482] T a469cf9350c14bdcbdf012fed32f42e3 P 1b5d8892eb234ddcb1e959d2c5617c2d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:44:08.066546 16797 raft_consensus.cc:3037] T a469cf9350c14bdcbdf012fed32f42e3 P 1b5d8892eb234ddcb1e959d2c5617c2d [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:44:08.067667 16797 raft_consensus.cc:504] T a469cf9350c14bdcbdf012fed32f42e3 P 1b5d8892eb234ddcb1e959d2c5617c2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db0770e11084629879664a73186272b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41783 } } peers { permanent_uuid: "1b5d8892eb234ddcb1e959d2c5617c2d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41007 } } peers { permanent_uuid: "313950c3398246619d5ca19a0372c094" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34813 } }
I20260927 12:44:08.068012 16797 leader_election.cc:288] T a469cf9350c14bdcbdf012fed32f42e3 P 1b5d8892eb234ddcb1e959d2c5617c2d [CANDIDATE]: Term 1 election: Requested vote from peers 9db0770e11084629879664a73186272b (127.0.98.3:41783), 313950c3398246619d5ca19a0372c094 (127.0.98.1:34813)
I20260927 12:44:08.068348 16733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a469cf9350c14bdcbdf012fed32f42e3" candidate_uuid: "1b5d8892eb234ddcb1e959d2c5617c2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9db0770e11084629879664a73186272b"
I20260927 12:44:08.068446 16463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a469cf9350c14bdcbdf012fed32f42e3" candidate_uuid: "1b5d8892eb234ddcb1e959d2c5617c2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "313950c3398246619d5ca19a0372c094"
I20260927 12:44:08.068506 16733 raft_consensus.cc:3037] T a469cf9350c14bdcbdf012fed32f42e3 P 9db0770e11084629879664a73186272b [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:44:08.068593 16463 raft_consensus.cc:3037] T a469cf9350c14bdcbdf012fed32f42e3 P 313950c3398246619d5ca19a0372c094 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:44:08.069621 16733 raft_consensus.cc:2454] T a469cf9350c14bdcbdf012fed32f42e3 P 9db0770e11084629879664a73186272b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b5d8892eb234ddcb1e959d2c5617c2d in term 1.
I20260927 12:44:08.069629 16463 raft_consensus.cc:2454] T a469cf9350c14bdcbdf012fed32f42e3 P 313950c3398246619d5ca19a0372c094 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b5d8892eb234ddcb1e959d2c5617c2d in term 1.
I20260927 12:44:08.069969 16534 leader_election.cc:302] T a469cf9350c14bdcbdf012fed32f42e3 P 1b5d8892eb234ddcb1e959d2c5617c2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b5d8892eb234ddcb1e959d2c5617c2d, 313950c3398246619d5ca19a0372c094; no voters:
I20260927 12:44:08.070175 16797 raft_consensus.cc:2781] T a469cf9350c14bdcbdf012fed32f42e3 P 1b5d8892eb234ddcb1e959d2c5617c2d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:44:08.070425 16797 raft_consensus.cc:686] T a469cf9350c14bdcbdf012fed32f42e3 P 1b5d8892eb234ddcb1e959d2c5617c2d [term 1 LEADER]: Becoming Leader. State: Replica: 1b5d8892eb234ddcb1e959d2c5617c2d, State: Running, Role: LEADER
I20260927 12:44:08.070802 16797 consensus_queue.cc:237] T a469cf9350c14bdcbdf012fed32f42e3 P 1b5d8892eb234ddcb1e959d2c5617c2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db0770e11084629879664a73186272b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41783 } } peers { permanent_uuid: "1b5d8892eb234ddcb1e959d2c5617c2d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41007 } } peers { permanent_uuid: "313950c3398246619d5ca19a0372c094" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34813 } }
W20260927 12:44:08.071834 16780 tablet.cc:2369] T a469cf9350c14bdcbdf012fed32f42e3 P 9db0770e11084629879664a73186272b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:44:08.074136 16320 catalog_manager.cc:5495] T a469cf9350c14bdcbdf012fed32f42e3 P 1b5d8892eb234ddcb1e959d2c5617c2d reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b5d8892eb234ddcb1e959d2c5617c2d (127.0.98.2). New cstate: current_term: 1 leader_uuid: "1b5d8892eb234ddcb1e959d2c5617c2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db0770e11084629879664a73186272b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b5d8892eb234ddcb1e959d2c5617c2d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "313950c3398246619d5ca19a0372c094" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34813 } health_report { overall_health: UNKNOWN } } }
I20260927 12:44:08.119881 16790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:08.116306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49978 (local address 127.0.98.2:41007)
0927 12:44:08.116441 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:08.116444 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:08.116457 (+ 13us) server_negotiation.cc:407] Connection header received
0927 12:44:08.116553 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:08.116555 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:08.116611 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:08.116701 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:44:08.116918 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:08.117614 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:08.118413 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:08.118627 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:08.118710 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:08.119075 (+ 365us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:44:08.119490 (+ 415us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:44:08.119585 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:08.119697 (+ 112us) server_negotiation.cc:299] Negotiation successful
0927 12:44:08.119749 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260927 12:44:08.125747 16648 tablet.cc:2369] T a469cf9350c14bdcbdf012fed32f42e3 P 1b5d8892eb234ddcb1e959d2c5617c2d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:44:08.126250 16733 raft_consensus.cc:1261] T a469cf9350c14bdcbdf012fed32f42e3 P 9db0770e11084629879664a73186272b [term 1 FOLLOWER]: Refusing update from remote peer 1b5d8892eb234ddcb1e959d2c5617c2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:44:08.126266 16463 raft_consensus.cc:1261] T a469cf9350c14bdcbdf012fed32f42e3 P 313950c3398246619d5ca19a0372c094 [term 1 FOLLOWER]: Refusing update from remote peer 1b5d8892eb234ddcb1e959d2c5617c2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:44:08.126924 16800 consensus_queue.cc:1035] T a469cf9350c14bdcbdf012fed32f42e3 P 1b5d8892eb234ddcb1e959d2c5617c2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "313950c3398246619d5ca19a0372c094" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:44:08.127116 16797 consensus_queue.cc:1035] T a469cf9350c14bdcbdf012fed32f42e3 P 1b5d8892eb234ddcb1e959d2c5617c2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9db0770e11084629879664a73186272b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 12:44:08.130224 16510 tablet.cc:2369] T a469cf9350c14bdcbdf012fed32f42e3 P 313950c3398246619d5ca19a0372c094: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:44:08.141695 16804 mvcc.cc:205] Tried to move back new op lower bound from 7333941445116780544 to 7333941444900585472. Current Snapshot: MvccSnapshot[applied={T|T < 7333941445116780544 or (T in {7333941445116780544})}]
I20260927 12:44:08.142866 16321 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:44:08.143047 16321 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:44:08.145654 16321 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6a84386bfb6447078525c886e771edd7: Sending DeleteTablet for 3 replicas of tablet a469cf9350c14bdcbdf012fed32f42e3
I20260927 12:44:08.146430 16443 tablet_service.cc:1514] Processing DeleteTablet for tablet a469cf9350c14bdcbdf012fed32f42e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:44:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42710
I20260927 12:44:08.146445 16713 tablet_service.cc:1514] Processing DeleteTablet for tablet a469cf9350c14bdcbdf012fed32f42e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:44:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50968
I20260927 12:44:08.146445 16581 tablet_service.cc:1514] Processing DeleteTablet for tablet a469cf9350c14bdcbdf012fed32f42e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:44:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49966
I20260927 12:44:08.146924 16812 tablet_replica.cc:330] T a469cf9350c14bdcbdf012fed32f42e3 P 313950c3398246619d5ca19a0372c094: stopping tablet replica
I20260927 12:44:08.147080 16812 raft_consensus.cc:2229] T a469cf9350c14bdcbdf012fed32f42e3 P 313950c3398246619d5ca19a0372c094 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:44:08.147221 16812 raft_consensus.cc:2258] T a469cf9350c14bdcbdf012fed32f42e3 P 313950c3398246619d5ca19a0372c094 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:44:08.147246 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 16359
I20260927 12:44:08.147349 16813 tablet_replica.cc:330] T a469cf9350c14bdcbdf012fed32f42e3 P 9db0770e11084629879664a73186272b: stopping tablet replica
I20260927 12:44:08.147683 16813 raft_consensus.cc:2229] T a469cf9350c14bdcbdf012fed32f42e3 P 9db0770e11084629879664a73186272b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:44:08.147243 16814 tablet_replica.cc:330] T a469cf9350c14bdcbdf012fed32f42e3 P 1b5d8892eb234ddcb1e959d2c5617c2d: stopping tablet replica
I20260927 12:44:08.148006 16814 raft_consensus.cc:2229] T a469cf9350c14bdcbdf012fed32f42e3 P 1b5d8892eb234ddcb1e959d2c5617c2d [term 1 LEADER]: Raft consensus shutting down.
I20260927 12:44:08.148324 16814 raft_consensus.cc:2258] T a469cf9350c14bdcbdf012fed32f42e3 P 1b5d8892eb234ddcb1e959d2c5617c2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:44:08.148777 16813 raft_consensus.cc:2258] T a469cf9350c14bdcbdf012fed32f42e3 P 9db0770e11084629879664a73186272b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:44:08.150264 16814 ts_tablet_manager.cc:1905] T a469cf9350c14bdcbdf012fed32f42e3 P 1b5d8892eb234ddcb1e959d2c5617c2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:44:08.150264 16813 ts_tablet_manager.cc:1905] T a469cf9350c14bdcbdf012fed32f42e3 P 9db0770e11084629879664a73186272b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:44:08.153144 16814 ts_tablet_manager.cc:1918] T a469cf9350c14bdcbdf012fed32f42e3 P 1b5d8892eb234ddcb1e959d2c5617c2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:44:08.153144 16813 ts_tablet_manager.cc:1918] T a469cf9350c14bdcbdf012fed32f42e3 P 9db0770e11084629879664a73186272b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:44:08.153250 16814 log.cc:1196] T a469cf9350c14bdcbdf012fed32f42e3 P 1b5d8892eb234ddcb1e959d2c5617c2d: Deleting WAL directory at /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-1/wal/wals/a469cf9350c14bdcbdf012fed32f42e3
I20260927 12:44:08.153368 16813 log.cc:1196] T a469cf9350c14bdcbdf012fed32f42e3 P 9db0770e11084629879664a73186272b: Deleting WAL directory at /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-2/wal/wals/a469cf9350c14bdcbdf012fed32f42e3
I20260927 12:44:08.153698 16814 ts_tablet_manager.cc:1939] T a469cf9350c14bdcbdf012fed32f42e3 P 1b5d8892eb234ddcb1e959d2c5617c2d: Deleting consensus metadata
I20260927 12:44:08.153843 16813 ts_tablet_manager.cc:1939] T a469cf9350c14bdcbdf012fed32f42e3 P 9db0770e11084629879664a73186272b: Deleting consensus metadata
I20260927 12:44:08.154945 16307 catalog_manager.cc:4841] TS 1b5d8892eb234ddcb1e959d2c5617c2d (127.0.98.2:41007): tablet a469cf9350c14bdcbdf012fed32f42e3 (table test-table [id=9345e991a9de42e9985923e1ff3141c8]) successfully deleted
I20260927 12:44:08.155174 16307 catalog_manager.cc:4841] TS 9db0770e11084629879664a73186272b (127.0.98.3:41783): tablet a469cf9350c14bdcbdf012fed32f42e3 (table test-table [id=9345e991a9de42e9985923e1ff3141c8]) successfully deleted
W20260927 12:44:08.157094 16305 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34813 (error 108)
I20260927 12:44:08.157979 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 16513
I20260927 12:44:08.158200 16789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:08.157881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34813 (local address 127.0.0.1:42714)
0927 12:44:08.158001 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:08.158096 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 12:44:08.159688 16305 catalog_manager.cc:4584] TS 313950c3398246619d5ca19a0372c094 (127.0.98.1:34813): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a469cf9350c14bdcbdf012fed32f42e3: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34813: connect: Connection refused (error 111)
I20260927 12:44:08.163543 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 16650
I20260927 12:44:08.169426 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 16288
I20260927 12:44:28.175158 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43175
--webserver_interface=127.0.98.62
--webserver_port=40691
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43175
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:44:28.289270 16816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:44:28.289680 16816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:44:28.289757 16816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:44:28.293887 16816 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 12:44:28.293962 16816 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 12:44:28.293987 16816 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 12:44:28.294013 16816 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:44:28.294051 16816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:44:28.294092 16816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:44:28.294128 16816 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:44:28.294163 16816 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:44:28.299657 16816 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43175
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=40691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:44:28.300782 16816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:44:28.301945 16816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:44:28.310706 16822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:44:28.310708 16824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:44:28.310708 16821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:44:28.311101 16816 server_base.cc:1034] running on GCE node
I20260927 12:44:28.311661 16816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:44:28.312871 16816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:44:28.314062 16816 hybrid_clock.cc:648] HybridClock initialized: now 1790513068314048 us; error 36 us; skew 500 ppm
Sep 27 12:44:28 dist-test-slave-bnc1 krb5kdc[15742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513068, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:44:28.320742 16816 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:44:28.321885 16816 webserver.cc:460] Webserver started at http://127.0.98.62:40691/ using document root <none> and password file <none>
I20260927 12:44:28.322472 16816 fs_manager.cc:359] Metadata directory not provided
I20260927 12:44:28.322566 16816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:44:28.326553 16816 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 12:44:28.328699 16831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:44:28.330009 16816 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:44:28.330147 16816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/master-0/wal
uuid: "6a84386bfb6447078525c886e771edd7"
format_stamp: "Formatted at 2026-09-27 12:43:45 on dist-test-slave-bnc1"
server_key: "fd8a4d08e16de8eba9ab390545f34ad1"
server_key_iv: "7d0076f3c5a54d20df6e644a4f337782"
server_key_version: "encryptionkey@0"
I20260927 12:44:28.330724 16816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:44:28.347003 16816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:44:28.350150 16816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:44:28.350358 16816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:44:28.357800 16816 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43175
I20260927 12:44:28.358099 16883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43175 every 8 connection(s)
I20260927 12:44:28.359122 16816 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:44:28.361984 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 16816
I20260927 12:44:28.362536 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:34813
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:43175
--webserver_port=41557
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 12:44:28.363534 16884 sys_catalog.cc:263] Verifying existing consensus state
I20260927 12:44:28.365191 16884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a84386bfb6447078525c886e771edd7: Bootstrap starting.
I20260927 12:44:28.382436 16884 log.cc:824] T 00000000000000000000000000000000 P 6a84386bfb6447078525c886e771edd7: Log is configured to *not* fsync() on all Append() calls
I20260927 12:44:28.393479 16884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a84386bfb6447078525c886e771edd7: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 12:44:28.394040 16884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a84386bfb6447078525c886e771edd7: Bootstrap complete.
I20260927 12:44:28.398255 16884 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a84386bfb6447078525c886e771edd7 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a84386bfb6447078525c886e771edd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43175 } }
I20260927 12:44:28.398665 16884 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a84386bfb6447078525c886e771edd7 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a84386bfb6447078525c886e771edd7, State: Initialized, Role: FOLLOWER
I20260927 12:44:28.399210 16884 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a84386bfb6447078525c886e771edd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a84386bfb6447078525c886e771edd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43175 } }
I20260927 12:44:28.399358 16884 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a84386bfb6447078525c886e771edd7 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:44:28.399449 16884 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a84386bfb6447078525c886e771edd7 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:44:28.399569 16884 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a84386bfb6447078525c886e771edd7 [term 2 FOLLOWER]: Advancing to term 3
I20260927 12:44:28.401501 16884 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a84386bfb6447078525c886e771edd7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a84386bfb6447078525c886e771edd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43175 } }
I20260927 12:44:28.401868 16884 leader_election.cc:302] T 00000000000000000000000000000000 P 6a84386bfb6447078525c886e771edd7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a84386bfb6447078525c886e771edd7; no voters:
I20260927 12:44:28.402225 16884 leader_election.cc:288] T 00000000000000000000000000000000 P 6a84386bfb6447078525c886e771edd7 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 12:44:28.402316 16888 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a84386bfb6447078525c886e771edd7 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 12:44:28.402557 16888 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a84386bfb6447078525c886e771edd7 [term 3 LEADER]: Becoming Leader. State: Replica: 6a84386bfb6447078525c886e771edd7, State: Running, Role: LEADER
I20260927 12:44:28.402933 16888 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a84386bfb6447078525c886e771edd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a84386bfb6447078525c886e771edd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43175 } }
I20260927 12:44:28.403517 16884 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a84386bfb6447078525c886e771edd7 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:44:28.404338 16889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a84386bfb6447078525c886e771edd7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "6a84386bfb6447078525c886e771edd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a84386bfb6447078525c886e771edd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43175 } } }
I20260927 12:44:28.404467 16889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a84386bfb6447078525c886e771edd7 [sys.catalog]: This master's current role is: LEADER
I20260927 12:44:28.404850 16890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a84386bfb6447078525c886e771edd7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a84386bfb6447078525c886e771edd7. Latest consensus state: current_term: 3 leader_uuid: "6a84386bfb6447078525c886e771edd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a84386bfb6447078525c886e771edd7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43175 } } }
I20260927 12:44:28.404986 16890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a84386bfb6447078525c886e771edd7 [sys.catalog]: This master's current role is: LEADER
I20260927 12:44:28.405278 16897 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:44:28.410789 16897 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:44:28.411406 16897 catalog_manager.cc:1259] Loaded cluster ID: e5734dd828784514a0003d1ff75ee734
I20260927 12:44:28.411465 16897 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:44:28.414280 16897 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:44:28.415010 16897 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6a84386bfb6447078525c886e771edd7: Loaded TSK: 0
I20260927 12:44:28.415094 16897 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6a84386bfb6447078525c886e771edd7: Loaded TSK: 1
F20260927 12:44:28.415819 16897 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f7025e729ad google::LogMessage::Fail() at ??:0
@ 0x7f7025e7588a google::LogMessage::SendToLog() at ??:0
@ 0x7f7025e724de google::LogMessage::Flush() at ??:0
@ 0x7f7025e73a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f7028b3618d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f7028b2ef0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f70307d77bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f70307b199c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f70307b2178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f70307f3df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f703081282b std::function<>::operator()() at ??:0
@ 0x7f70307b1b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f70307b2f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f70307af579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f70307f3640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f70280c43d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f70280c4c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f70280c63f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f70280b5512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f70296f96db start_thread at ??:0
@ 0x7f702497061f clone at ??:0
W20260927 12:44:28.501499 16886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:44:28.501799 16886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:44:28.501842 16886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:44:28.506224 16886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:44:28.506271 16886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:44:28.506330 16886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:44:28.512159 16886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:34813
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=41557
--tserver_master_addrs=127.0.98.62:43175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16886
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:44:28.513132 16886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:44:28.514165 16886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:44:28.527680 16931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:44:28.527858 16932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:44:28.528595 16934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:44:28.528859 16886 server_base.cc:1034] running on GCE node
I20260927 12:44:28.529320 16886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:44:28.530046 16886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:44:28.531230 16886 hybrid_clock.cc:648] HybridClock initialized: now 1790513068531192 us; error 56 us; skew 500 ppm
Sep 27 12:44:28 dist-test-slave-bnc1 krb5kdc[15742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513068, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:44:28.534198 16886 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 12:44:28.535112 16886 webserver.cc:460] Webserver started at http://127.0.98.1:41557/ using document root <none> and password file <none>
I20260927 12:44:28.535681 16886 fs_manager.cc:359] Metadata directory not provided
I20260927 12:44:28.535740 16886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:44:28.539680 16886 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 12:44:28.542012 16941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:44:28.543327 16886 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:44:28.543466 16886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-0/wal
uuid: "313950c3398246619d5ca19a0372c094"
format_stamp: "Formatted at 2026-09-27 12:43:46 on dist-test-slave-bnc1"
server_key: "305ae662cc990c826d7cfb53b629b7eb"
server_key_iv: "1faf95d7b50307bb97c15d9fbe413e22"
server_key_version: "encryptionkey@0"
I20260927 12:44:28.544027 16886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:44:28.555930 16886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:44:28.559038 16886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:44:28.559244 16886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:44:28.559809 16886 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:44:28.560853 16948 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260927 12:44:28.566206 16886 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 12:44:28.566282 16886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260927 12:44:28.566366 16886 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 12:44:28.567930 16886 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 12:44:28.568001 16886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260927 12:44:28.568298 16948 tablet_bootstrap.cc:492] T a469cf9350c14bdcbdf012fed32f42e3 P 313950c3398246619d5ca19a0372c094: Bootstrap starting.
I20260927 12:44:28.578639 16886 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34813
I20260927 12:44:28.578689 17055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34813 every 8 connection(s)
I20260927 12:44:28.579576 16886 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:44:28.589461 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 16886
I20260927 12:44:28.589958 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:41007
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:43175
--webserver_port=38755
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 12:44:28.590112 16948 log.cc:824] T a469cf9350c14bdcbdf012fed32f42e3 P 313950c3398246619d5ca19a0372c094: Log is configured to *not* fsync() on all Append() calls
I20260927 12:44:28.599757 16948 tablet_bootstrap.cc:492] T a469cf9350c14bdcbdf012fed32f42e3 P 313950c3398246619d5ca19a0372c094: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 12:44:28.600225 16948 tablet_bootstrap.cc:492] T a469cf9350c14bdcbdf012fed32f42e3 P 313950c3398246619d5ca19a0372c094: Bootstrap complete.
I20260927 12:44:28.601755 16948 ts_tablet_manager.cc:1397] T a469cf9350c14bdcbdf012fed32f42e3 P 313950c3398246619d5ca19a0372c094: Time spent bootstrapping tablet: real 0.034s user 0.019s sys 0.014s
I20260927 12:44:28.605098 16948 raft_consensus.cc:348] T a469cf9350c14bdcbdf012fed32f42e3 P 313950c3398246619d5ca19a0372c094 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db0770e11084629879664a73186272b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41783 } } peers { permanent_uuid: "1b5d8892eb234ddcb1e959d2c5617c2d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41007 } } peers { permanent_uuid: "313950c3398246619d5ca19a0372c094" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34813 } }
I20260927 12:44:28.605711 16948 raft_consensus.cc:729] T a469cf9350c14bdcbdf012fed32f42e3 P 313950c3398246619d5ca19a0372c094 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 313950c3398246619d5ca19a0372c094, State: Initialized, Role: FOLLOWER
I20260927 12:44:28.606240 16948 consensus_queue.cc:260] T a469cf9350c14bdcbdf012fed32f42e3 P 313950c3398246619d5ca19a0372c094 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db0770e11084629879664a73186272b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41783 } } peers { permanent_uuid: "1b5d8892eb234ddcb1e959d2c5617c2d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41007 } } peers { permanent_uuid: "313950c3398246619d5ca19a0372c094" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34813 } }
I20260927 12:44:28.607285 16948 ts_tablet_manager.cc:1428] T a469cf9350c14bdcbdf012fed32f42e3 P 313950c3398246619d5ca19a0372c094: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
W20260927 12:44:28.671451 17058 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:44:28.581383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43175 (local address 127.0.98.1:58277)
0927 12:44:28.581853 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:28.581905 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 12:44:28.582672 (+ 767us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:44:28.670403 (+ 87731us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43175: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":149,"threads_started":1}
W20260927 12:44:28.672150 17056 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43175 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:43175: Client connection negotiation failed: client connection to 127.0.98.62:43175: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 12:44:28.672945 17058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:28.672639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43175 (local address 127.0.98.1:33939)
0927 12:44:28.672804 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:28.672882 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260927 12:44:28.673650 17058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:28.673392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43175 (local address 127.0.98.1:47419)
0927 12:44:28.673495 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:28.673541 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 12:44:28.673815 17056 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 12:44:28.713349 17060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:44:28.713722 17060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:44:28.713810 17060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:44:28.717994 17060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:44:28.718060 17060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:44:28.718155 17060 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:44:28.723714 17060 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:41007
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=38755
--tserver_master_addrs=127.0.98.62:43175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17060
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:44:28.724795 17060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:44:28.725858 17060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:44:28.735522 17069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:44:28.735532 17067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:44:28.735777 17060 server_base.cc:1034] running on GCE node
W20260927 12:44:28.735534 17066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:44:28.736254 17060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:44:28.736908 17060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:44:28.738091 17060 hybrid_clock.cc:648] HybridClock initialized: now 1790513068738070 us; error 39 us; skew 500 ppm
Sep 27 12:44:28 dist-test-slave-bnc1 krb5kdc[15742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513068, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:44:28.740821 17060 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 12:44:28.741760 17060 webserver.cc:460] Webserver started at http://127.0.98.2:38755/ using document root <none> and password file <none>
I20260927 12:44:28.742282 17060 fs_manager.cc:359] Metadata directory not provided
I20260927 12:44:28.742340 17060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:44:28.746232 17060 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 12:44:28.749070 17076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:44:28.750497 17060 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:44:28.750604 17060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-1/wal
uuid: "1b5d8892eb234ddcb1e959d2c5617c2d"
format_stamp: "Formatted at 2026-09-27 12:43:46 on dist-test-slave-bnc1"
server_key: "be8c5a9d34a8afa35d67ab16958fd8c0"
server_key_iv: "dac9f3589347e525b205bf19d941c5a8"
server_key_version: "encryptionkey@0"
I20260927 12:44:28.751164 17060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:44:28.782006 17060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:44:28.785506 17060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:44:28.785733 17060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:44:28.786304 17060 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:44:28.787101 17060 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:44:28.787148 17060 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:44:28.787189 17060 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:44:28.787204 17060 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:44:28.796693 17060 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41007
I20260927 12:44:28.796723 17189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41007 every 8 connection(s)
I20260927 12:44:28.797698 17060 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:44:28.798168 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 17060
I20260927 12:44:28.798664 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:41783
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:43175
--webserver_port=45361
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 12:44:28.802063 17193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:28.800209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43175 (local address 127.0.98.2:59165)
0927 12:44:28.801098 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:28.801245 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":665,"thread_start_us":536,"threads_started":1}
W20260927 12:44:28.802560 17190 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43175 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:43175: Client connection negotiation failed: client connection to 127.0.98.62:43175: connect: Connection refused (error 111)
I20260927 12:44:28.803176 17193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:28.802955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43175 (local address 127.0.98.2:48891)
0927 12:44:28.803064 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:28.803112 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260927 12:44:28.803836 17193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:28.803629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43175 (local address 127.0.98.2:59935)
0927 12:44:28.803732 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:28.803792 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 12:44:28.804047 17190 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 12:44:28.916775 17192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:44:28.917069 17192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:44:28.917131 17192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:44:28.921103 17192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:44:28.921183 17192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:44:28.921243 17192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:44:28.926753 17192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:41783
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=45361
--tserver_master_addrs=127.0.98.62:43175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:44:28.927784 17192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:44:28.928761 17192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:44:28.938020 17201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:44:28.938086 17199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:44:28.938107 17198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:44:28.938263 17192 server_base.cc:1034] running on GCE node
I20260927 12:44:28.938665 17192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:44:28.939276 17192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:44:28.940462 17192 hybrid_clock.cc:648] HybridClock initialized: now 1790513068940447 us; error 28 us; skew 500 ppm
Sep 27 12:44:28 dist-test-slave-bnc1 krb5kdc[15742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513068, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:44:28.943213 17192 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 12:44:28.944115 17192 webserver.cc:460] Webserver started at http://127.0.98.3:45361/ using document root <none> and password file <none>
I20260927 12:44:28.944703 17192 fs_manager.cc:359] Metadata directory not provided
I20260927 12:44:28.944828 17192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:44:28.948823 17192 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 12:44:28.951015 17208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:44:28.952286 17192 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 12:44:28.952427 17192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-2/wal
uuid: "9db0770e11084629879664a73186272b"
format_stamp: "Formatted at 2026-09-27 12:43:46 on dist-test-slave-bnc1"
server_key: "36222f23d19619549387b908d09b3d7c"
server_key_iv: "9cc54157df03191e36fcd45ff88b1bf2"
server_key_version: "encryptionkey@0"
I20260927 12:44:28.952994 17192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:44:28.974450 17192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:44:28.978014 17192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:44:28.978338 17192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:44:28.978982 17192 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:44:28.979844 17192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:44:28.979916 17192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:44:28.979985 17192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:44:28.980032 17192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:44:28.989508 17192 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41783
I20260927 12:44:28.989526 17321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41783 every 8 connection(s)
I20260927 12:44:28.990567 17192 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:44:28.994251 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:28.992691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43175 (local address 127.0.98.3:57117)
0927 12:44:28.993347 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:28.993455 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":107,"threads_started":1}
W20260927 12:44:28.994705 17322 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43175 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:43175: Client connection negotiation failed: client connection to 127.0.98.62:43175: connect: Connection refused (error 111)
I20260927 12:44:28.995339 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:28.995109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43175 (local address 127.0.98.3:52449)
0927 12:44:28.995221 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:28.995267 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 12:44:28.995841 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 17192
I20260927 12:44:28.996037 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:28.995800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43175 (local address 127.0.98.3:40827)
0927 12:44:28.995923 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:28.995978 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 12:44:28.996208 17322 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 12:44:29.674960 17326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:29.674501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43175 (local address 127.0.98.1:33405)
0927 12:44:29.674771 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:29.674850 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":82,"threads_started":1}
I20260927 12:44:29.805110 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:29.804676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43175 (local address 127.0.98.2:33685)
0927 12:44:29.804930 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:29.805006 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":66,"threads_started":1}
I20260927 12:44:29.955972 17328 raft_consensus.cc:482] T a469cf9350c14bdcbdf012fed32f42e3 P 313950c3398246619d5ca19a0372c094 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:44:29.956200 17328 raft_consensus.cc:504] T a469cf9350c14bdcbdf012fed32f42e3 P 313950c3398246619d5ca19a0372c094 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db0770e11084629879664a73186272b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41783 } } peers { permanent_uuid: "1b5d8892eb234ddcb1e959d2c5617c2d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41007 } } peers { permanent_uuid: "313950c3398246619d5ca19a0372c094" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34813 } }
I20260927 12:44:29.957417 17328 leader_election.cc:288] T a469cf9350c14bdcbdf012fed32f42e3 P 313950c3398246619d5ca19a0372c094 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9db0770e11084629879664a73186272b (127.0.98.3:41783), 1b5d8892eb234ddcb1e959d2c5617c2d (127.0.98.2:41007)
Sep 27 12:44:29 dist-test-slave-bnc1 krb5kdc[15742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513068, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Sep 27 12:44:29 dist-test-slave-bnc1 krb5kdc[15742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513068, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260927 12:44:29.972761 17326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:29.957692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41783 (local address 127.0.98.1:36785)
0927 12:44:29.957843 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:29.957860 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 12:44:29.958015 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:44:29.959819 (+ 1804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:44:29.959826 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:44:29.960433 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:44:29.960915 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:29.960930 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:29.962571 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:29.962576 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:29.963447 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:29.963458 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:29.964218 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:29.964222 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:29.964420 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:29.965067 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:44:29.965084 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:44:29.967291 (+ 2207us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:44:29.969553 (+ 2262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:44:29.969560 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:44:29.969589 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:44:29.969977 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:44:29.970411 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:44:29.970415 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:44:29.970417 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:44:29.970720 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:44:29.971501 (+ 781us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:44:29.971504 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:44:29.971892 (+ 388us) client_negotiation.cc:769] Sending connection context
0927 12:44:29.972117 (+ 225us) client_negotiation.cc:240] Negotiation successful
0927 12:44:29.972531 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63,"mutex_wait_us":231,"spinlock_wait_cycles":32896}
I20260927 12:44:29.972918 17331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:29.958414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52275 (local address 127.0.98.2:41007)
0927 12:44:29.958695 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:29.958715 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:29.958747 (+ 32us) server_negotiation.cc:407] Connection header received
0927 12:44:29.959464 (+ 717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:29.959469 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:29.959859 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:29.960171 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:44:29.961094 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:29.962301 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:29.963622 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:29.964084 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:29.964265 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:29.967128 (+ 2863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:44:29.967133 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:44:29.967145 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:44:29.967171 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:44:29.969267 (+ 2096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:44:29.970108 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:44:29.970114 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:44:29.970120 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:44:29.970281 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:44:29.970828 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:44:29.970831 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:44:29.970833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:44:29.971351 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:44:29.971487 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:29.972556 (+ 1069us) server_negotiation.cc:299] Negotiation successful
0927 12:44:29.972786 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":103,"threads_started":1}
I20260927 12:44:29.972761 17330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:29.957911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41007 (local address 127.0.98.1:52275)
0927 12:44:29.958218 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:29.958232 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:44:29.958317 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:44:29.960124 (+ 1807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:44:29.960126 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:44:29.960418 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:44:29.960911 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:29.960926 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:29.962451 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:29.962454 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:29.963445 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:29.963455 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:29.964212 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:29.964216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:29.964420 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:29.965130 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:44:29.965142 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:44:29.966909 (+ 1767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:44:29.969407 (+ 2498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:44:29.969412 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:44:29.969424 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:44:29.969977 (+ 553us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:44:29.970411 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:44:29.970415 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:44:29.970417 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:44:29.970720 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:44:29.971488 (+ 768us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:44:29.971496 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:44:29.971892 (+ 396us) client_negotiation.cc:769] Sending connection context
0927 12:44:29.972117 (+ 225us) client_negotiation.cc:240] Negotiation successful
0927 12:44:29.972588 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"mutex_wait_us":310,"spinlock_wait_cycles":15744,"thread_start_us":111,"threads_started":1}
I20260927 12:44:29.973603 17329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:29.957837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36785 (local address 127.0.98.3:41783)
0927 12:44:29.958216 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:29.958238 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:29.958277 (+ 39us) server_negotiation.cc:407] Connection header received
0927 12:44:29.959066 (+ 789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:29.959072 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:29.959437 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:29.959691 (+ 254us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:44:29.961154 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:29.962412 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:29.963593 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:29.964085 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:29.964277 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:29.967489 (+ 3212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:44:29.967500 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:44:29.967510 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:44:29.967540 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:44:29.969438 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:44:29.970108 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:44:29.970114 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:44:29.970119 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:44:29.970281 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:44:29.970834 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:44:29.970837 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:44:29.970839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:44:29.971351 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:44:29.971487 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:29.973207 (+ 1720us) server_negotiation.cc:299] Negotiation successful
0927 12:44:29.973416 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":126,"threads_started":1}
W20260927 12:44:29.975757 16944 leader_election.cc:341] T a469cf9350c14bdcbdf012fed32f42e3 P 313950c3398246619d5ca19a0372c094 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1b5d8892eb234ddcb1e959d2c5617c2d (127.0.98.2:41007): Not found: Tablet not found: a469cf9350c14bdcbdf012fed32f42e3
W20260927 12:44:29.975998 16944 leader_election.cc:341] T a469cf9350c14bdcbdf012fed32f42e3 P 313950c3398246619d5ca19a0372c094 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9db0770e11084629879664a73186272b (127.0.98.3:41783): Not found: Tablet not found: a469cf9350c14bdcbdf012fed32f42e3
I20260927 12:44:29.976084 16944 leader_election.cc:302] T a469cf9350c14bdcbdf012fed32f42e3 P 313950c3398246619d5ca19a0372c094 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 313950c3398246619d5ca19a0372c094; no voters: 1b5d8892eb234ddcb1e959d2c5617c2d, 9db0770e11084629879664a73186272b
I20260927 12:44:29.976372 17328 raft_consensus.cc:2726] T a469cf9350c14bdcbdf012fed32f42e3 P 313950c3398246619d5ca19a0372c094 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 12:44:29.997095 17332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:29.996732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43175 (local address 127.0.98.3:34577)
0927 12:44:29.996958 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:29.997007 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":65,"threads_started":1}
I20260927 12:44:30.676213 17333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:30.675740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43175 (local address 127.0.98.1:44907)
0927 12:44:30.676035 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:30.676098 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":80,"threads_started":1}
I20260927 12:44:30.806424 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:30.806015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43175 (local address 127.0.98.2:57769)
0927 12:44:30.806267 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:30.806322 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":85,"threads_started":1}
I20260927 12:44:30.998248 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:30.997849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43175 (local address 127.0.98.3:53681)
0927 12:44:30.998085 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:30.998145 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":73,"threads_started":1}
I20260927 12:44:31.677449 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:31.677021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43175 (local address 127.0.98.1:43693)
0927 12:44:31.677289 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:31.677349 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":106,"threads_started":1}
I20260927 12:44:31.807518 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:31.807119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43175 (local address 127.0.98.2:32975)
0927 12:44:31.807353 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:31.807409 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":75,"threads_started":1}
I20260927 12:44:31.999747 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:31.999320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43175 (local address 127.0.98.3:38311)
0927 12:44:31.999588 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:31.999653 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":79,"threads_started":1}
I20260927 12:44:32.133773 17339 raft_consensus.cc:482] T a469cf9350c14bdcbdf012fed32f42e3 P 313950c3398246619d5ca19a0372c094 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:44:32.133920 17339 raft_consensus.cc:504] T a469cf9350c14bdcbdf012fed32f42e3 P 313950c3398246619d5ca19a0372c094 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db0770e11084629879664a73186272b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41783 } } peers { permanent_uuid: "1b5d8892eb234ddcb1e959d2c5617c2d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41007 } } peers { permanent_uuid: "313950c3398246619d5ca19a0372c094" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34813 } }
I20260927 12:44:32.134321 17339 leader_election.cc:288] T a469cf9350c14bdcbdf012fed32f42e3 P 313950c3398246619d5ca19a0372c094 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9db0770e11084629879664a73186272b (127.0.98.3:41783), 1b5d8892eb234ddcb1e959d2c5617c2d (127.0.98.2:41007)
W20260927 12:44:32.135234 16944 leader_election.cc:341] T a469cf9350c14bdcbdf012fed32f42e3 P 313950c3398246619d5ca19a0372c094 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9db0770e11084629879664a73186272b (127.0.98.3:41783): Not found: Tablet not found: a469cf9350c14bdcbdf012fed32f42e3
W20260927 12:44:32.135411 16944 leader_election.cc:341] T a469cf9350c14bdcbdf012fed32f42e3 P 313950c3398246619d5ca19a0372c094 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1b5d8892eb234ddcb1e959d2c5617c2d (127.0.98.2:41007): Not found: Tablet not found: a469cf9350c14bdcbdf012fed32f42e3
I20260927 12:44:32.135464 16944 leader_election.cc:302] T a469cf9350c14bdcbdf012fed32f42e3 P 313950c3398246619d5ca19a0372c094 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 313950c3398246619d5ca19a0372c094; no voters: 1b5d8892eb234ddcb1e959d2c5617c2d, 9db0770e11084629879664a73186272b
I20260927 12:44:32.135654 17339 raft_consensus.cc:2726] T a469cf9350c14bdcbdf012fed32f42e3 P 313950c3398246619d5ca19a0372c094 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 12:44:32.678649 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:32.678195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43175 (local address 127.0.98.1:40959)
0927 12:44:32.678483 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:32.678544 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":71,"threads_started":1}
I20260927 12:44:32.808707 17341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:32.808277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43175 (local address 127.0.98.2:34853)
0927 12:44:32.808565 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:32.808618 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":126,"threads_started":1}
I20260927 12:44:33.000952 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:33.000522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43175 (local address 127.0.98.3:55011)
0927 12:44:33.000776 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:33.000835 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":74,"threads_started":1}
I20260927 12:44:33.679943 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:33.679503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43175 (local address 127.0.98.1:37567)
0927 12:44:33.679786 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:33.679842 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":85,"threads_started":1}
I20260927 12:44:33.809945 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:33.809486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43175 (local address 127.0.98.2:51343)
0927 12:44:33.809779 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:33.809851 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":131,"threads_started":1}
I20260927 12:44:34.002296 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:34.001861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43175 (local address 127.0.98.3:33933)
0927 12:44:34.002140 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:34.002203 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":87,"threads_started":1}
I20260927 12:44:34.681079 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:34.680690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43175 (local address 127.0.98.1:42747)
0927 12:44:34.680908 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:34.680958 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":71,"threads_started":1}
I20260927 12:44:34.810971 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:34.810604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43175 (local address 127.0.98.2:52181)
0927 12:44:34.810826 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:34.810875 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":72,"threads_started":1}
I20260927 12:44:35.003625 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:35.003160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43175 (local address 127.0.98.3:52395)
0927 12:44:35.003464 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:35.003529 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":82,"threads_started":1}
I20260927 12:44:35.241894 17349 raft_consensus.cc:482] T a469cf9350c14bdcbdf012fed32f42e3 P 313950c3398246619d5ca19a0372c094 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:44:35.242030 17349 raft_consensus.cc:504] T a469cf9350c14bdcbdf012fed32f42e3 P 313950c3398246619d5ca19a0372c094 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db0770e11084629879664a73186272b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41783 } } peers { permanent_uuid: "1b5d8892eb234ddcb1e959d2c5617c2d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41007 } } peers { permanent_uuid: "313950c3398246619d5ca19a0372c094" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34813 } }
I20260927 12:44:35.242564 17349 leader_election.cc:288] T a469cf9350c14bdcbdf012fed32f42e3 P 313950c3398246619d5ca19a0372c094 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9db0770e11084629879664a73186272b (127.0.98.3:41783), 1b5d8892eb234ddcb1e959d2c5617c2d (127.0.98.2:41007)
W20260927 12:44:35.243414 16944 leader_election.cc:341] T a469cf9350c14bdcbdf012fed32f42e3 P 313950c3398246619d5ca19a0372c094 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9db0770e11084629879664a73186272b (127.0.98.3:41783): Not found: Tablet not found: a469cf9350c14bdcbdf012fed32f42e3
W20260927 12:44:35.243558 16944 leader_election.cc:341] T a469cf9350c14bdcbdf012fed32f42e3 P 313950c3398246619d5ca19a0372c094 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1b5d8892eb234ddcb1e959d2c5617c2d (127.0.98.2:41007): Not found: Tablet not found: a469cf9350c14bdcbdf012fed32f42e3
I20260927 12:44:35.243614 16944 leader_election.cc:302] T a469cf9350c14bdcbdf012fed32f42e3 P 313950c3398246619d5ca19a0372c094 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 313950c3398246619d5ca19a0372c094; no voters: 1b5d8892eb234ddcb1e959d2c5617c2d, 9db0770e11084629879664a73186272b
I20260927 12:44:35.243773 17349 raft_consensus.cc:2726] T a469cf9350c14bdcbdf012fed32f42e3 P 313950c3398246619d5ca19a0372c094 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 12:44:35.682294 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:35.681868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43175 (local address 127.0.98.1:57501)
0927 12:44:35.682133 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:35.682191 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":85,"threads_started":1}
I20260927 12:44:35.812126 17351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:35.811699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43175 (local address 127.0.98.2:38819)
0927 12:44:35.811980 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:35.812036 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":79,"threads_started":1}
I20260927 12:44:36.005084 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:36.004553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43175 (local address 127.0.98.3:33033)
0927 12:44:36.004909 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:36.004981 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":89,"threads_started":1}
I20260927 12:44:36.683539 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:36.683155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43175 (local address 127.0.98.1:52365)
0927 12:44:36.683392 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:36.683446 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":74,"threads_started":1}
I20260927 12:44:36.813344 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:36.812974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43175 (local address 127.0.98.2:55679)
0927 12:44:36.813197 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:36.813250 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":71,"threads_started":1}
I20260927 12:44:36.985721 17355 raft_consensus.cc:482] T a469cf9350c14bdcbdf012fed32f42e3 P 313950c3398246619d5ca19a0372c094 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:44:36.985857 17355 raft_consensus.cc:504] T a469cf9350c14bdcbdf012fed32f42e3 P 313950c3398246619d5ca19a0372c094 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db0770e11084629879664a73186272b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41783 } } peers { permanent_uuid: "1b5d8892eb234ddcb1e959d2c5617c2d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41007 } } peers { permanent_uuid: "313950c3398246619d5ca19a0372c094" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34813 } }
I20260927 12:44:36.986284 17355 leader_election.cc:288] T a469cf9350c14bdcbdf012fed32f42e3 P 313950c3398246619d5ca19a0372c094 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9db0770e11084629879664a73186272b (127.0.98.3:41783), 1b5d8892eb234ddcb1e959d2c5617c2d (127.0.98.2:41007)
W20260927 12:44:36.987054 16944 leader_election.cc:341] T a469cf9350c14bdcbdf012fed32f42e3 P 313950c3398246619d5ca19a0372c094 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1b5d8892eb234ddcb1e959d2c5617c2d (127.0.98.2:41007): Not found: Tablet not found: a469cf9350c14bdcbdf012fed32f42e3
W20260927 12:44:36.987174 16944 leader_election.cc:341] T a469cf9350c14bdcbdf012fed32f42e3 P 313950c3398246619d5ca19a0372c094 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9db0770e11084629879664a73186272b (127.0.98.3:41783): Not found: Tablet not found: a469cf9350c14bdcbdf012fed32f42e3
I20260927 12:44:36.987227 16944 leader_election.cc:302] T a469cf9350c14bdcbdf012fed32f42e3 P 313950c3398246619d5ca19a0372c094 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 313950c3398246619d5ca19a0372c094; no voters: 1b5d8892eb234ddcb1e959d2c5617c2d, 9db0770e11084629879664a73186272b
I20260927 12:44:36.987388 17355 raft_consensus.cc:2726] T a469cf9350c14bdcbdf012fed32f42e3 P 313950c3398246619d5ca19a0372c094 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 12:44:37.006330 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:37.005938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43175 (local address 127.0.98.3:44879)
0927 12:44:37.006182 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:37.006234 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":87,"threads_started":1}
I20260927 12:44:37.684715 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:37.684300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43175 (local address 127.0.98.1:41473)
0927 12:44:37.684569 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:37.684622 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":89,"threads_started":1}
I20260927 12:44:37.814508 17358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:37.814101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43175 (local address 127.0.98.2:50571)
0927 12:44:37.814358 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:37.814413 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":88,"threads_started":1}
I20260927 12:44:38.007562 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:38.007137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43175 (local address 127.0.98.3:54229)
0927 12:44:38.007403 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:38.007463 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":93,"threads_started":1}
I20260927 12:44:38.686180 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:38.685752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43175 (local address 127.0.98.1:47569)
0927 12:44:38.686024 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:38.686084 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":91,"threads_started":1}
I20260927 12:44:38.815696 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:38.815298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43175 (local address 127.0.98.2:49085)
0927 12:44:38.815542 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:38.815601 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":74,"threads_started":1}
I20260927 12:44:39.008781 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:39.008311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43175 (local address 127.0.98.3:52077)
0927 12:44:39.008626 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:39.008688 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":76,"threads_started":1}
I20260927 12:44:39.477102 17363 raft_consensus.cc:482] T a469cf9350c14bdcbdf012fed32f42e3 P 313950c3398246619d5ca19a0372c094 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:44:39.477262 17363 raft_consensus.cc:504] T a469cf9350c14bdcbdf012fed32f42e3 P 313950c3398246619d5ca19a0372c094 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9db0770e11084629879664a73186272b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41783 } } peers { permanent_uuid: "1b5d8892eb234ddcb1e959d2c5617c2d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41007 } } peers { permanent_uuid: "313950c3398246619d5ca19a0372c094" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34813 } }
I20260927 12:44:39.477742 17363 leader_election.cc:288] T a469cf9350c14bdcbdf012fed32f42e3 P 313950c3398246619d5ca19a0372c094 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9db0770e11084629879664a73186272b (127.0.98.3:41783), 1b5d8892eb234ddcb1e959d2c5617c2d (127.0.98.2:41007)
W20260927 12:44:39.478580 16944 leader_election.cc:341] T a469cf9350c14bdcbdf012fed32f42e3 P 313950c3398246619d5ca19a0372c094 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1b5d8892eb234ddcb1e959d2c5617c2d (127.0.98.2:41007): Not found: Tablet not found: a469cf9350c14bdcbdf012fed32f42e3
W20260927 12:44:39.478727 16944 leader_election.cc:341] T a469cf9350c14bdcbdf012fed32f42e3 P 313950c3398246619d5ca19a0372c094 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9db0770e11084629879664a73186272b (127.0.98.3:41783): Not found: Tablet not found: a469cf9350c14bdcbdf012fed32f42e3
I20260927 12:44:39.478785 16944 leader_election.cc:302] T a469cf9350c14bdcbdf012fed32f42e3 P 313950c3398246619d5ca19a0372c094 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 313950c3398246619d5ca19a0372c094; no voters: 1b5d8892eb234ddcb1e959d2c5617c2d, 9db0770e11084629879664a73186272b
I20260927 12:44:39.478943 17363 raft_consensus.cc:2726] T a469cf9350c14bdcbdf012fed32f42e3 P 313950c3398246619d5ca19a0372c094 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 12:44:39.687292 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:39.686903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43175 (local address 127.0.98.1:52233)
0927 12:44:39.687144 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:39.687198 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":83,"threads_started":1}
I20260927 12:44:39.816918 17365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:39.816501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43175 (local address 127.0.98.2:46283)
0927 12:44:39.816767 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:39.816822 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":89,"threads_started":1}
I20260927 12:44:40.010097 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:40.009647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43175 (local address 127.0.98.3:44941)
0927 12:44:40.009946 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:40.010005 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":97,"threads_started":1}
I20260927 12:44:40.688495 17367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:40.688080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43175 (local address 127.0.98.1:44703)
0927 12:44:40.688336 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:40.688401 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":99,"threads_started":1}
I20260927 12:44:40.818298 17368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:40.817855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43175 (local address 127.0.98.2:43901)
0927 12:44:40.818131 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:40.818195 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":92,"threads_started":1}
I20260927 12:44:41.011303 17369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:41.010853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43175 (local address 127.0.98.3:35759)
0927 12:44:41.011126 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:41.011192 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":88,"threads_started":1}
I20260927 12:44:41.689860 17370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:41.689374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43175 (local address 127.0.98.1:37475)
0927 12:44:41.689702 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:41.689761 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":78,"threads_started":1}
I20260927 12:44:41.819621 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:41.819144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43175 (local address 127.0.98.2:45483)
0927 12:44:41.819455 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:41.819522 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":76,"threads_started":1}
I20260927 12:44:42.012727 17372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:42.012260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43175 (local address 127.0.98.3:45057)
0927 12:44:42.012570 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:42.012631 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":142,"threads_started":1}
I20260927 12:44:42.691179 17373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:42.690706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43175 (local address 127.0.98.1:51535)
0927 12:44:42.691010 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:42.691082 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":129,"threads_started":1}
I20260927 12:44:42.821094 17374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:42.820567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43175 (local address 127.0.98.2:39337)
0927 12:44:42.820922 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:42.820992 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":84,"threads_started":1}
I20260927 12:44:43.014142 17375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:43.013685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43175 (local address 127.0.98.3:44753)
0927 12:44:43.013979 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:43.014044 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":99,"threads_started":1}
I20260927 12:44:43.692689 17376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:43.692229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43175 (local address 127.0.98.1:41773)
0927 12:44:43.692522 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:43.692590 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":70,"threads_started":1}
I20260927 12:44:43.822651 17377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:43.822212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43175 (local address 127.0.98.2:52123)
0927 12:44:43.822495 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:43.822556 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":81,"threads_started":1}
I20260927 12:44:44.002628 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 16886
I20260927 12:44:44.008777 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 17060
I20260927 12:44:44.015014 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 17192
2026-09-27T12:44:44Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59551 ms)
[----------] 30 tests from SecurityITest (243167 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:44:44 dist-test-slave-bnc1 krb5kdc[17380](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:44:44 dist-test-slave-bnc1 krb5kdc[17380](info): set up 2 sockets
Sep 27 12:44:44 dist-test-slave-bnc1 krb5kdc[17380](info): commencing operation
krb5kdc: starting...
W20260927 12:44:45.053485 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/krb5kdc/test-user.keytab.
Sep 27 12:44:45 dist-test-slave-bnc1 krb5kdc[17380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:44:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:44:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:44:45.178645 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37047
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37047
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:44:45.287508 17393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:44:45.287813 17393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:44:45.287892 17393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:44:45.291941 17393 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:44:45.292030 17393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:44:45.292069 17393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:44:45.292129 17393 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:44:45.292182 17393 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:44:45.297360 17393 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37047
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17393
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:44:45.298481 17393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:44:45.299515 17393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:44:45.308523 17401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:44:45.308676 17393 server_base.cc:1034] running on GCE node
W20260927 12:44:45.308641 17399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:44:45.308557 17398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:44:45.309427 17393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:44:45.310571 17393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:44:45.311760 17393 hybrid_clock.cc:648] HybridClock initialized: now 1790513085311746 us; error 26 us; skew 500 ppm
Sep 27 12:44:45 dist-test-slave-bnc1 krb5kdc[17380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513085, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:44:45.314687 17393 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:44:45.315593 17393 webserver.cc:460] Webserver started at http://127.0.98.62:33793/ using document root <none> and password file <none>
I20260927 12:44:45.316149 17393 fs_manager.cc:359] Metadata directory not provided
I20260927 12:44:45.316267 17393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:44:45.316488 17393 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:44:45.318580 17393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/minicluster-data/master-0/data/instance:
uuid: "798b2baecc6c4090bff49464ca14c5db"
format_stamp: "Formatted at 2026-09-27 12:44:45 on dist-test-slave-bnc1"
server_key: "aed7068337187e4afda44b7fc22c2b2a"
server_key_iv: "f85c601e2ba34e2c8d1a32f995063bf4"
server_key_version: "encryptionkey@0"
I20260927 12:44:45.319133 17393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/minicluster-data/master-0/wal/instance:
uuid: "798b2baecc6c4090bff49464ca14c5db"
format_stamp: "Formatted at 2026-09-27 12:44:45 on dist-test-slave-bnc1"
server_key: "aed7068337187e4afda44b7fc22c2b2a"
server_key_iv: "f85c601e2ba34e2c8d1a32f995063bf4"
server_key_version: "encryptionkey@0"
I20260927 12:44:45.322777 17393 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 12:44:45.325172 17408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:44:45.326385 17393 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 12:44:45.326525 17393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/minicluster-data/master-0/wal
uuid: "798b2baecc6c4090bff49464ca14c5db"
format_stamp: "Formatted at 2026-09-27 12:44:45 on dist-test-slave-bnc1"
server_key: "aed7068337187e4afda44b7fc22c2b2a"
server_key_iv: "f85c601e2ba34e2c8d1a32f995063bf4"
server_key_version: "encryptionkey@0"
I20260927 12:44:45.326637 17393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:44:45.356063 17393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:44:45.359632 17393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:44:45.359838 17393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:44:45.367077 17393 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37047
I20260927 12:44:45.367101 17460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37047 every 8 connection(s)
I20260927 12:44:45.368012 17393 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:44:45.370827 17461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:44:45.375345 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 17393
I20260927 12:44:45.375455 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/minicluster-data/master-0/wal/instance
I20260927 12:44:45.375756 392 external_mini_cluster.cc:1428] Setting key 84fd2ca91d325460d78e6155e8060100
I20260927 12:44:45.376766 17461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 798b2baecc6c4090bff49464ca14c5db: Bootstrap starting.
I20260927 12:44:45.379235 17461 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 798b2baecc6c4090bff49464ca14c5db: Neither blocks nor log segments found. Creating new log.
I20260927 12:44:45.380019 17461 log.cc:824] T 00000000000000000000000000000000 P 798b2baecc6c4090bff49464ca14c5db: Log is configured to *not* fsync() on all Append() calls
I20260927 12:44:45.382021 17461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 798b2baecc6c4090bff49464ca14c5db: No bootstrap required, opened a new log
Sep 27 12:44:45 dist-test-slave-bnc1 krb5kdc[17380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:44:45.385903 17461 raft_consensus.cc:348] T 00000000000000000000000000000000 P 798b2baecc6c4090bff49464ca14c5db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "798b2baecc6c4090bff49464ca14c5db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37047 } }
I20260927 12:44:45.386162 17461 raft_consensus.cc:374] T 00000000000000000000000000000000 P 798b2baecc6c4090bff49464ca14c5db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:44:45.386238 17461 raft_consensus.cc:729] T 00000000000000000000000000000000 P 798b2baecc6c4090bff49464ca14c5db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 798b2baecc6c4090bff49464ca14c5db, State: Initialized, Role: FOLLOWER
I20260927 12:44:45.386729 17461 consensus_queue.cc:260] T 00000000000000000000000000000000 P 798b2baecc6c4090bff49464ca14c5db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "798b2baecc6c4090bff49464ca14c5db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37047 } }
I20260927 12:44:45.386890 17461 raft_consensus.cc:388] T 00000000000000000000000000000000 P 798b2baecc6c4090bff49464ca14c5db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:44:45.386972 17461 raft_consensus.cc:482] T 00000000000000000000000000000000 P 798b2baecc6c4090bff49464ca14c5db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:44:45.387085 17461 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 798b2baecc6c4090bff49464ca14c5db [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:44:45.388007 17461 raft_consensus.cc:504] T 00000000000000000000000000000000 P 798b2baecc6c4090bff49464ca14c5db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "798b2baecc6c4090bff49464ca14c5db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37047 } }
I20260927 12:44:45.388360 17461 leader_election.cc:302] T 00000000000000000000000000000000 P 798b2baecc6c4090bff49464ca14c5db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 798b2baecc6c4090bff49464ca14c5db; no voters:
I20260927 12:44:45.388675 17461 leader_election.cc:288] T 00000000000000000000000000000000 P 798b2baecc6c4090bff49464ca14c5db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:44:45.388789 17466 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 798b2baecc6c4090bff49464ca14c5db [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:44:45.389009 17466 raft_consensus.cc:686] T 00000000000000000000000000000000 P 798b2baecc6c4090bff49464ca14c5db [term 1 LEADER]: Becoming Leader. State: Replica: 798b2baecc6c4090bff49464ca14c5db, State: Running, Role: LEADER
I20260927 12:44:45.389402 17466 consensus_queue.cc:237] T 00000000000000000000000000000000 P 798b2baecc6c4090bff49464ca14c5db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "798b2baecc6c4090bff49464ca14c5db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37047 } }
I20260927 12:44:45.390079 17461 sys_catalog.cc:564] T 00000000000000000000000000000000 P 798b2baecc6c4090bff49464ca14c5db [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:44:45.391108 17464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:45.377202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43592 (local address 127.0.98.62:37047)
0927 12:44:45.377708 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:45.377725 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:45.377768 (+ 43us) server_negotiation.cc:407] Connection header received
0927 12:44:45.378417 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:45.378434 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:45.378786 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:45.379138 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:44:45.379983 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:45.381029 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:45.381943 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:45.382488 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:45.382659 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:45.385476 (+ 2817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:44:45.385481 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:44:45.385495 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:44:45.385522 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:44:45.387635 (+ 2113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:44:45.388075 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:44:45.388081 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:44:45.388087 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:44:45.388229 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:44:45.388527 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:44:45.388530 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:44:45.388532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:44:45.389137 (+ 605us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:44:45.389335 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:45.389757 (+ 422us) server_negotiation.cc:299] Negotiation successful
0927 12:44:45.390050 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":141,"threads_started":1}
I20260927 12:44:45.391072 17467 sys_catalog.cc:455] T 00000000000000000000000000000000 P 798b2baecc6c4090bff49464ca14c5db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "798b2baecc6c4090bff49464ca14c5db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "798b2baecc6c4090bff49464ca14c5db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37047 } } }
I20260927 12:44:45.391192 17467 sys_catalog.cc:458] T 00000000000000000000000000000000 P 798b2baecc6c4090bff49464ca14c5db [sys.catalog]: This master's current role is: LEADER
I20260927 12:44:45.391531 17468 sys_catalog.cc:455] T 00000000000000000000000000000000 P 798b2baecc6c4090bff49464ca14c5db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 798b2baecc6c4090bff49464ca14c5db. Latest consensus state: current_term: 1 leader_uuid: "798b2baecc6c4090bff49464ca14c5db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "798b2baecc6c4090bff49464ca14c5db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37047 } } }
I20260927 12:44:45.391630 17468 sys_catalog.cc:458] T 00000000000000000000000000000000 P 798b2baecc6c4090bff49464ca14c5db [sys.catalog]: This master's current role is: LEADER
I20260927 12:44:45.391919 17475 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:44:45.395578 17475 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:44:45.401343 17475 catalog_manager.cc:1347] Generated new cluster ID: 11283dd94ab74d3497470e29c414a859
I20260927 12:44:45.401407 17475 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:44:45.411368 17475 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:44:45.412629 17475 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:44:45.423507 17475 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 798b2baecc6c4090bff49464ca14c5db: Generated new TSK 0
I20260927 12:44:45.424230 17475 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 12:44:45.454942 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37047
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:44:45.572341 17487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:44:45.572631 17487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:44:45.572691 17487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:44:45.577554 17487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:44:45.577651 17487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:44:45.577775 17487 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:44:45.583194 17487 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37047
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:44:45.584254 17487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:44:45.585294 17487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:44:45.595068 17492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:44:45.595036 17493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:44:45.595129 17487 server_base.cc:1034] running on GCE node
W20260927 12:44:45.595022 17495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:44:45.595706 17487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:44:45.596329 17487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:44:45.597517 17487 hybrid_clock.cc:648] HybridClock initialized: now 1790513085597500 us; error 31 us; skew 500 ppm
Sep 27 12:44:45 dist-test-slave-bnc1 krb5kdc[17380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513085, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:44:45.600499 17487 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 12:44:45.601449 17487 webserver.cc:460] Webserver started at http://127.0.98.1:44689/ using document root <none> and password file <none>
I20260927 12:44:45.602047 17487 fs_manager.cc:359] Metadata directory not provided
I20260927 12:44:45.602113 17487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:44:45.602370 17487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:44:45.604485 17487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/minicluster-data/ts-0/data/instance:
uuid: "a5187e9123af4d1eaadab092a3f72576"
format_stamp: "Formatted at 2026-09-27 12:44:45 on dist-test-slave-bnc1"
server_key: "d12f03417cde366303e0b5100466b11a"
server_key_iv: "0b0db430f5f9c3755963c933a34adc3d"
server_key_version: "encryptionkey@0"
I20260927 12:44:45.605070 17487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a5187e9123af4d1eaadab092a3f72576"
format_stamp: "Formatted at 2026-09-27 12:44:45 on dist-test-slave-bnc1"
server_key: "d12f03417cde366303e0b5100466b11a"
server_key_iv: "0b0db430f5f9c3755963c933a34adc3d"
server_key_version: "encryptionkey@0"
I20260927 12:44:45.609031 17487 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 12:44:45.611709 17502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:44:45.612996 17487 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:44:45.613130 17487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/minicluster-data/ts-0/wal
uuid: "a5187e9123af4d1eaadab092a3f72576"
format_stamp: "Formatted at 2026-09-27 12:44:45 on dist-test-slave-bnc1"
server_key: "d12f03417cde366303e0b5100466b11a"
server_key_iv: "0b0db430f5f9c3755963c933a34adc3d"
server_key_version: "encryptionkey@0"
I20260927 12:44:45.613277 17487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:44:45.638892 17487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:44:45.642755 17487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:44:45.642982 17487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:44:45.643647 17487 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:44:45.644548 17487 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:44:45.644623 17487 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:44:45.644696 17487 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:44:45.644753 17487 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:44:45.654585 17487 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33507
I20260927 12:44:45.654608 17615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33507 every 8 connection(s)
I20260927 12:44:45.655517 17487 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:44:45.661849 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 17487
I20260927 12:44:45.661983 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/minicluster-data/ts-0/wal/instance
I20260927 12:44:45.662350 392 external_mini_cluster.cc:1428] Setting key fb05296b56f41c4929ca9f3a2e4c9b30
Sep 27 12:44:45 dist-test-slave-bnc1 krb5kdc[17380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513085, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:44:45.670706 17464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:45.657404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43827 (local address 127.0.98.62:37047)
0927 12:44:45.657562 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:45.657584 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:45.658502 (+ 918us) server_negotiation.cc:407] Connection header received
0927 12:44:45.659819 (+ 1317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:45.659821 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:45.659877 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:45.659955 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:44:45.661324 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:45.661961 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:45.663590 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:45.663907 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:45.664026 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:45.666572 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:44:45.666575 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:44:45.666578 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:44:45.666602 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:44:45.668159 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:44:45.668824 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:44:45.668826 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:44:45.668827 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:44:45.668899 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:44:45.669370 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:44:45.669374 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:44:45.669375 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:44:45.669650 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:44:45.669767 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:45.670457 (+ 690us) server_negotiation.cc:299] Negotiation successful
0927 12:44:45.670571 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 12:44:45.671963 17618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:45.657840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37047 (local address 127.0.98.1:43827)
0927 12:44:45.658344 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:45.658388 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 12:44:45.659615 (+ 1227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:44:45.660124 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:44:45.660131 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:44:45.660632 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:44:45.661171 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:45.661186 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:45.662106 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:45.662110 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:45.663464 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:45.663475 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:45.664065 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:45.664069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:45.664251 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:45.664872 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:44:45.664888 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:44:45.666400 (+ 1512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:44:45.668285 (+ 1885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:44:45.668294 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:44:45.668313 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:44:45.668704 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:44:45.668994 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:44:45.668999 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:44:45.669005 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:44:45.669237 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:44:45.669807 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:44:45.669815 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:44:45.670157 (+ 342us) client_negotiation.cc:769] Sending connection context
0927 12:44:45.670388 (+ 231us) client_negotiation.cc:240] Negotiation successful
0927 12:44:45.670776 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":115,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 12:44:45.673292 17616 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37047
I20260927 12:44:45.673542 17616 heartbeater.cc:464] Registering TS with master...
I20260927 12:44:45.674145 17616 heartbeater.cc:511] Master 127.0.98.62:37047 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 12:44:45.675895 17425 ts_manager.cc:194] Registered new tserver with Master: a5187e9123af4d1eaadab092a3f72576 (127.0.98.1:33507)
I20260927 12:44:45.677453 17425 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43827
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 12:44:45.691269 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37047
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:44:45.803916 17621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:44:45.804190 17621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:44:45.804234 17621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:44:45.808307 17621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:44:45.808370 17621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:44:45.808430 17621 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:44:45.813684 17621 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37047
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:44:45.814661 17621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:44:45.815599 17621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:44:45.825374 17627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:44:45.825371 17629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:44:45.825485 17621 server_base.cc:1034] running on GCE node
W20260927 12:44:45.825397 17626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:44:45.826148 17621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:44:45.826821 17621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:44:45.828014 17621 hybrid_clock.cc:648] HybridClock initialized: now 1790513085827980 us; error 46 us; skew 500 ppm
Sep 27 12:44:45 dist-test-slave-bnc1 krb5kdc[17380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513085, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:44:45.830730 17621 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 12:44:45.831630 17621 webserver.cc:460] Webserver started at http://127.0.98.2:37873/ using document root <none> and password file <none>
I20260927 12:44:45.832144 17621 fs_manager.cc:359] Metadata directory not provided
I20260927 12:44:45.832201 17621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:44:45.832373 17621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:44:45.834470 17621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/minicluster-data/ts-1/data/instance:
uuid: "60502677e1b44304a598a89d364e31ee"
format_stamp: "Formatted at 2026-09-27 12:44:45 on dist-test-slave-bnc1"
server_key: "f7646ec282f184238d0f1e2f62b473ff"
server_key_iv: "9de3be37a479d1fff62ce9ca4884044e"
server_key_version: "encryptionkey@0"
I20260927 12:44:45.834981 17621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/minicluster-data/ts-1/wal/instance:
uuid: "60502677e1b44304a598a89d364e31ee"
format_stamp: "Formatted at 2026-09-27 12:44:45 on dist-test-slave-bnc1"
server_key: "f7646ec282f184238d0f1e2f62b473ff"
server_key_iv: "9de3be37a479d1fff62ce9ca4884044e"
server_key_version: "encryptionkey@0"
I20260927 12:44:45.838677 17621 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 12:44:45.841061 17636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:44:45.842367 17621 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 12:44:45.842473 17621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/minicluster-data/ts-1/wal
uuid: "60502677e1b44304a598a89d364e31ee"
format_stamp: "Formatted at 2026-09-27 12:44:45 on dist-test-slave-bnc1"
server_key: "f7646ec282f184238d0f1e2f62b473ff"
server_key_iv: "9de3be37a479d1fff62ce9ca4884044e"
server_key_version: "encryptionkey@0"
I20260927 12:44:45.842566 17621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:44:45.855199 17621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:44:45.858248 17621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:44:45.858400 17621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:44:45.858919 17621 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:44:45.859680 17621 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:44:45.859726 17621 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:44:45.859766 17621 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:44:45.859781 17621 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:44:45.869151 17621 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40551
I20260927 12:44:45.869174 17749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40551 every 8 connection(s)
I20260927 12:44:45.870103 17621 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:44:45.877884 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 17621
I20260927 12:44:45.878028 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/minicluster-data/ts-1/wal/instance
I20260927 12:44:45.878386 392 external_mini_cluster.cc:1428] Setting key dd4e44e8a8dbae09a7253405489e59d5
Sep 27 12:44:45 dist-test-slave-bnc1 krb5kdc[17380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513085, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:44:45.886232 17464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:45.872216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42621 (local address 127.0.98.62:37047)
0927 12:44:45.872384 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:45.872390 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:45.873336 (+ 946us) server_negotiation.cc:407] Connection header received
0927 12:44:45.874750 (+ 1414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:45.874753 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:45.874806 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:45.874898 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:44:45.876139 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:45.876752 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:45.878103 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:45.878440 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:45.878549 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:45.881839 (+ 3290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:44:45.881846 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:44:45.881851 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:44:45.881887 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:44:45.883728 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:44:45.884265 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:44:45.884273 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:44:45.884278 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:44:45.884384 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:44:45.884809 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:44:45.884813 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:44:45.884819 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:44:45.885084 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:44:45.885199 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:45.885776 (+ 577us) server_negotiation.cc:299] Negotiation successful
0927 12:44:45.885944 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 12:44:45.886830 17752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:45.872687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37047 (local address 127.0.98.2:42621)
0927 12:44:45.873151 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:45.873197 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 12:44:45.874491 (+ 1294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:44:45.875062 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:44:45.875069 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:44:45.875448 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:44:45.875973 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:45.875987 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:45.876871 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:45.876874 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:45.877946 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:45.877958 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:45.878611 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:45.878614 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:45.878830 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:45.879566 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:44:45.879583 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:44:45.881609 (+ 2026us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:44:45.883861 (+ 2252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:44:45.883865 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:44:45.883878 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:44:45.884139 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:44:45.884482 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:44:45.884484 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:44:45.884485 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:44:45.884689 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:44:45.885195 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:44:45.885201 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:44:45.885444 (+ 243us) client_negotiation.cc:769] Sending connection context
0927 12:44:45.885670 (+ 226us) client_negotiation.cc:240] Negotiation successful
0927 12:44:45.885955 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":128,"threads_started":1}
I20260927 12:44:45.888123 17750 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37047
I20260927 12:44:45.888417 17750 heartbeater.cc:464] Registering TS with master...
I20260927 12:44:45.889130 17750 heartbeater.cc:511] Master 127.0.98.62:37047 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 12:44:45.890457 17425 ts_manager.cc:194] Registered new tserver with Master: 60502677e1b44304a598a89d364e31ee (127.0.98.2:40551)
I20260927 12:44:45.891204 17425 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42621
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 12:44:45.908499 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37047
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:44:46.022864 17755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:44:46.023123 17755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:44:46.023205 17755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:44:46.027319 17755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:44:46.027388 17755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:44:46.027479 17755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:44:46.033123 17755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37047
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17755
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:44:46.034307 17755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:44:46.035364 17755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:44:46.045174 17761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:44:46.045454 17760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:44:46.045428 17763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:44:46.045748 17755 server_base.cc:1034] running on GCE node
I20260927 12:44:46.046092 17755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:44:46.046831 17755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:44:46.047996 17755 hybrid_clock.cc:648] HybridClock initialized: now 1790513086047962 us; error 46 us; skew 500 ppm
Sep 27 12:44:46 dist-test-slave-bnc1 krb5kdc[17380](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513086, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:44:46.050858 17755 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 12:44:46.051692 17755 webserver.cc:460] Webserver started at http://127.0.98.3:42573/ using document root <none> and password file <none>
I20260927 12:44:46.052270 17755 fs_manager.cc:359] Metadata directory not provided
I20260927 12:44:46.052361 17755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:44:46.052584 17755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:44:46.054697 17755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/minicluster-data/ts-2/data/instance:
uuid: "892c00d9cd8a47b2b1121ca2b59df6ba"
format_stamp: "Formatted at 2026-09-27 12:44:46 on dist-test-slave-bnc1"
server_key: "357e1b6c482bfde41890d62337423cfd"
server_key_iv: "9e1159da6a090d9baf930296ea912b6d"
server_key_version: "encryptionkey@0"
I20260927 12:44:46.055261 17755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/minicluster-data/ts-2/wal/instance:
uuid: "892c00d9cd8a47b2b1121ca2b59df6ba"
format_stamp: "Formatted at 2026-09-27 12:44:46 on dist-test-slave-bnc1"
server_key: "357e1b6c482bfde41890d62337423cfd"
server_key_iv: "9e1159da6a090d9baf930296ea912b6d"
server_key_version: "encryptionkey@0"
I20260927 12:44:46.059043 17755 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 12:44:46.061493 17770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:44:46.062853 17755 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 12:44:46.062985 17755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/minicluster-data/ts-2/wal
uuid: "892c00d9cd8a47b2b1121ca2b59df6ba"
format_stamp: "Formatted at 2026-09-27 12:44:46 on dist-test-slave-bnc1"
server_key: "357e1b6c482bfde41890d62337423cfd"
server_key_iv: "9e1159da6a090d9baf930296ea912b6d"
server_key_version: "encryptionkey@0"
I20260927 12:44:46.063097 17755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:44:46.074993 17755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:44:46.078094 17755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:44:46.078294 17755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:44:46.078891 17755 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:44:46.079710 17755 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:44:46.079779 17755 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:44:46.079872 17755 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:44:46.079916 17755 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:44:46.088824 17755 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33001
I20260927 12:44:46.088850 17883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33001 every 8 connection(s)
I20260927 12:44:46.089820 17755 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:44:46.094908 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 17755
I20260927 12:44:46.095075 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/minicluster-data/ts-2/wal/instance
I20260927 12:44:46.095428 392 external_mini_cluster.cc:1428] Setting key 1f5431466201d7ce32bafc091d6816d7
Sep 27 12:44:46 dist-test-slave-bnc1 krb5kdc[17380](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513086, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:44:46.104900 17464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:46.091924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56125 (local address 127.0.98.62:37047)
0927 12:44:46.092055 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:46.092060 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:46.093004 (+ 944us) server_negotiation.cc:407] Connection header received
0927 12:44:46.094426 (+ 1422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:46.094429 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:46.094474 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:46.094549 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:44:46.096269 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:46.097047 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:46.098432 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:46.098646 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:46.098728 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:46.101387 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:44:46.101390 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:44:46.101392 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:44:46.101416 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:44:46.102967 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:44:46.103494 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:44:46.103497 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:44:46.103498 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:44:46.103571 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:44:46.103890 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:44:46.103893 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:44:46.103894 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:44:46.104070 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:44:46.104145 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:46.104666 (+ 521us) server_negotiation.cc:299] Negotiation successful
0927 12:44:46.104766 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 12:44:46.105988 17886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:46.092393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37047 (local address 127.0.98.3:56125)
0927 12:44:46.092848 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:46.092894 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 12:44:46.094202 (+ 1308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:44:46.094795 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:44:46.094805 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:44:46.095400 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:44:46.096106 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:46.096130 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:46.097173 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:46.097176 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:46.098328 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:46.098338 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:46.098778 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:46.098781 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:46.098923 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:46.099607 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:44:46.099625 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:44:46.101215 (+ 1590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:44:46.103078 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:44:46.103083 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:44:46.103096 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:44:46.103407 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:44:46.103654 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:44:46.103656 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:44:46.103657 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:44:46.103808 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:44:46.104161 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:44:46.104166 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:44:46.104451 (+ 285us) client_negotiation.cc:769] Sending connection context
0927 12:44:46.104626 (+ 175us) client_negotiation.cc:240] Negotiation successful
0927 12:44:46.104912 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":113,"threads_started":1}
I20260927 12:44:46.107187 17884 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37047
I20260927 12:44:46.107460 17884 heartbeater.cc:464] Registering TS with master...
I20260927 12:44:46.108052 17884 heartbeater.cc:511] Master 127.0.98.62:37047 requested a full tablet report, sending...
I20260927 12:44:46.109216 17425 ts_manager.cc:194] Registered new tserver with Master: 892c00d9cd8a47b2b1121ca2b59df6ba (127.0.98.3:33001)
I20260927 12:44:46.109879 17425 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56125
I20260927 12:44:46.121790 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 12:44:46.132421 17464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:46.123811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43600 (local address 127.0.98.62:37047)
0927 12:44:46.123943 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:46.123946 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:46.124214 (+ 268us) server_negotiation.cc:407] Connection header received
0927 12:44:46.124362 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:46.124365 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:46.124413 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:46.124496 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:44:46.125404 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:46.125951 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:46.127016 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:46.127269 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:46.127363 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:46.128761 (+ 1398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:44:46.128764 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:44:46.128767 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:44:46.128793 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:44:46.130517 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:44:46.131029 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:44:46.131038 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:44:46.131039 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:44:46.131159 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:44:46.131453 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:44:46.131458 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:44:46.131459 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:44:46.131718 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:44:46.131868 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:46.132110 (+ 242us) server_negotiation.cc:299] Negotiation successful
0927 12:44:46.132243 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 12:44:46.136010 17425 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43600:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 12:44:46.139436 17425 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:44:46.154803 17899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:46.148507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33001 (local address 127.0.0.1:58120)
0927 12:44:46.149452 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:46.149507 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 12:44:46.149769 (+ 262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:44:46.150369 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:44:46.150375 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:44:46.150398 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:44:46.150568 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:46.150576 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:46.151823 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:46.151829 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:46.153134 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:46.153144 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:46.154241 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:46.154245 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:46.154350 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:46.154390 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 12:44:46.154514 (+ 124us) client_negotiation.cc:240] Negotiation successful
0927 12:44:46.154639 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":770,"thread_start_us":95,"threads_started":1}
I20260927 12:44:46.154863 17898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:46.148796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58120 (local address 127.0.98.3:33001)
0927 12:44:46.149385 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:46.149400 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:46.149644 (+ 244us) server_negotiation.cc:407] Connection header received
0927 12:44:46.149972 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:46.149979 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:46.150201 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:46.150358 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:44:46.150710 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:46.151674 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:46.153278 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:46.154094 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:46.154216 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:46.154383 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:46.154560 (+ 177us) server_negotiation.cc:299] Negotiation successful
0927 12:44:46.154670 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":108,"threads_started":1}
I20260927 12:44:46.155159 17897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:46.148507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40551 (local address 127.0.0.1:59826)
0927 12:44:46.149786 (+ 1279us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:46.149800 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:44:46.149874 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:44:46.150554 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:44:46.150558 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:44:46.150590 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:44:46.150883 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:46.150891 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:46.152271 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:46.152273 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:46.153639 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:46.153656 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:46.154741 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:46.154744 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:46.154847 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:46.154879 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 12:44:46.154956 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 12:44:46.155037 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1221,"mutex_wait_us":756,"spinlock_wait_cycles":3584,"thread_start_us":104,"threads_started":1}
I20260927 12:44:46.155510 17896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:46.148764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59826 (local address 127.0.98.2:40551)
0927 12:44:46.150099 (+ 1335us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:46.150110 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:46.150130 (+ 20us) server_negotiation.cc:407] Connection header received
0927 12:44:46.150215 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:46.150222 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:46.150404 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:46.150675 (+ 271us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:44:46.151017 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:46.152141 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:46.153795 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:46.154448 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:46.154542 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:46.154682 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:46.155244 (+ 562us) server_negotiation.cc:299] Negotiation successful
0927 12:44:46.155348 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1241,"thread_start_us":110,"threads_started":1}
I20260927 12:44:46.155656 17895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:46.148797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33507 (local address 127.0.0.1:51810)
0927 12:44:46.149684 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:46.149700 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 12:44:46.149811 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:44:46.150555 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:44:46.150558 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:44:46.150588 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:44:46.150741 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:46.150749 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:46.152268 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:46.152272 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:46.153647 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:46.153654 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:46.155019 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:46.155023 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:46.155145 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:46.155166 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 12:44:46.155267 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 12:44:46.155496 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":823,"mutex_wait_us":201,"spinlock_wait_cycles":5888,"thread_start_us":83,"threads_started":1}
I20260927 12:44:46.156312 17900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:46.149392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51810 (local address 127.0.98.1:33507)
0927 12:44:46.150085 (+ 693us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:46.150102 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:46.150125 (+ 23us) server_negotiation.cc:407] Connection header received
0927 12:44:46.150214 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:46.150218 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:46.150427 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:46.150592 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:44:46.150875 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:46.152097 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:46.153778 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:46.154668 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:46.154796 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:46.154941 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:46.156033 (+ 1092us) server_negotiation.cc:299] Negotiation successful
0927 12:44:46.156146 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":125,"threads_started":1}
I20260927 12:44:46.158409 17684 tablet_service.cc:1467] Processing CreateTablet for tablet b0d7752eb0574dd1a86c55f68fa47b68 (DEFAULT_TABLE table=test-table [id=2ad7ebbda50d465daffd7901ac51996d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:44:46.158409 17818 tablet_service.cc:1467] Processing CreateTablet for tablet b0d7752eb0574dd1a86c55f68fa47b68 (DEFAULT_TABLE table=test-table [id=2ad7ebbda50d465daffd7901ac51996d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:44:46.158802 17550 tablet_service.cc:1467] Processing CreateTablet for tablet b0d7752eb0574dd1a86c55f68fa47b68 (DEFAULT_TABLE table=test-table [id=2ad7ebbda50d465daffd7901ac51996d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:44:46.159533 17684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0d7752eb0574dd1a86c55f68fa47b68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:44:46.159533 17818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0d7752eb0574dd1a86c55f68fa47b68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:44:46.159754 17550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0d7752eb0574dd1a86c55f68fa47b68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:44:46.166296 17901 tablet_bootstrap.cc:492] T b0d7752eb0574dd1a86c55f68fa47b68 P a5187e9123af4d1eaadab092a3f72576: Bootstrap starting.
I20260927 12:44:46.167505 17903 tablet_bootstrap.cc:492] T b0d7752eb0574dd1a86c55f68fa47b68 P 892c00d9cd8a47b2b1121ca2b59df6ba: Bootstrap starting.
I20260927 12:44:46.167693 17902 tablet_bootstrap.cc:492] T b0d7752eb0574dd1a86c55f68fa47b68 P 60502677e1b44304a598a89d364e31ee: Bootstrap starting.
I20260927 12:44:46.168747 17901 tablet_bootstrap.cc:654] T b0d7752eb0574dd1a86c55f68fa47b68 P a5187e9123af4d1eaadab092a3f72576: Neither blocks nor log segments found. Creating new log.
I20260927 12:44:46.169646 17903 tablet_bootstrap.cc:654] T b0d7752eb0574dd1a86c55f68fa47b68 P 892c00d9cd8a47b2b1121ca2b59df6ba: Neither blocks nor log segments found. Creating new log.
I20260927 12:44:46.170096 17902 tablet_bootstrap.cc:654] T b0d7752eb0574dd1a86c55f68fa47b68 P 60502677e1b44304a598a89d364e31ee: Neither blocks nor log segments found. Creating new log.
I20260927 12:44:46.170073 17901 log.cc:824] T b0d7752eb0574dd1a86c55f68fa47b68 P a5187e9123af4d1eaadab092a3f72576: Log is configured to *not* fsync() on all Append() calls
I20260927 12:44:46.170998 17902 log.cc:824] T b0d7752eb0574dd1a86c55f68fa47b68 P 60502677e1b44304a598a89d364e31ee: Log is configured to *not* fsync() on all Append() calls
I20260927 12:44:46.170998 17903 log.cc:824] T b0d7752eb0574dd1a86c55f68fa47b68 P 892c00d9cd8a47b2b1121ca2b59df6ba: Log is configured to *not* fsync() on all Append() calls
I20260927 12:44:46.172137 17901 tablet_bootstrap.cc:492] T b0d7752eb0574dd1a86c55f68fa47b68 P a5187e9123af4d1eaadab092a3f72576: No bootstrap required, opened a new log
I20260927 12:44:46.172363 17901 ts_tablet_manager.cc:1397] T b0d7752eb0574dd1a86c55f68fa47b68 P a5187e9123af4d1eaadab092a3f72576: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 12:44:46.172967 17903 tablet_bootstrap.cc:492] T b0d7752eb0574dd1a86c55f68fa47b68 P 892c00d9cd8a47b2b1121ca2b59df6ba: No bootstrap required, opened a new log
I20260927 12:44:46.172959 17902 tablet_bootstrap.cc:492] T b0d7752eb0574dd1a86c55f68fa47b68 P 60502677e1b44304a598a89d364e31ee: No bootstrap required, opened a new log
I20260927 12:44:46.173141 17902 ts_tablet_manager.cc:1397] T b0d7752eb0574dd1a86c55f68fa47b68 P 60502677e1b44304a598a89d364e31ee: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260927 12:44:46.173141 17903 ts_tablet_manager.cc:1397] T b0d7752eb0574dd1a86c55f68fa47b68 P 892c00d9cd8a47b2b1121ca2b59df6ba: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260927 12:44:46.175930 17901 raft_consensus.cc:348] T b0d7752eb0574dd1a86c55f68fa47b68 P a5187e9123af4d1eaadab092a3f72576 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "892c00d9cd8a47b2b1121ca2b59df6ba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33001 } } peers { permanent_uuid: "60502677e1b44304a598a89d364e31ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40551 } } peers { permanent_uuid: "a5187e9123af4d1eaadab092a3f72576" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33507 } }
I20260927 12:44:46.176195 17901 raft_consensus.cc:374] T b0d7752eb0574dd1a86c55f68fa47b68 P a5187e9123af4d1eaadab092a3f72576 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:44:46.176285 17901 raft_consensus.cc:729] T b0d7752eb0574dd1a86c55f68fa47b68 P a5187e9123af4d1eaadab092a3f72576 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5187e9123af4d1eaadab092a3f72576, State: Initialized, Role: FOLLOWER
I20260927 12:44:46.177022 17901 consensus_queue.cc:260] T b0d7752eb0574dd1a86c55f68fa47b68 P a5187e9123af4d1eaadab092a3f72576 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "892c00d9cd8a47b2b1121ca2b59df6ba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33001 } } peers { permanent_uuid: "60502677e1b44304a598a89d364e31ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40551 } } peers { permanent_uuid: "a5187e9123af4d1eaadab092a3f72576" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33507 } }
I20260927 12:44:46.177029 17903 raft_consensus.cc:348] T b0d7752eb0574dd1a86c55f68fa47b68 P 892c00d9cd8a47b2b1121ca2b59df6ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "892c00d9cd8a47b2b1121ca2b59df6ba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33001 } } peers { permanent_uuid: "60502677e1b44304a598a89d364e31ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40551 } } peers { permanent_uuid: "a5187e9123af4d1eaadab092a3f72576" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33507 } }
I20260927 12:44:46.177029 17902 raft_consensus.cc:348] T b0d7752eb0574dd1a86c55f68fa47b68 P 60502677e1b44304a598a89d364e31ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "892c00d9cd8a47b2b1121ca2b59df6ba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33001 } } peers { permanent_uuid: "60502677e1b44304a598a89d364e31ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40551 } } peers { permanent_uuid: "a5187e9123af4d1eaadab092a3f72576" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33507 } }
I20260927 12:44:46.177290 17902 raft_consensus.cc:374] T b0d7752eb0574dd1a86c55f68fa47b68 P 60502677e1b44304a598a89d364e31ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:44:46.177321 17903 raft_consensus.cc:374] T b0d7752eb0574dd1a86c55f68fa47b68 P 892c00d9cd8a47b2b1121ca2b59df6ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:44:46.177379 17902 raft_consensus.cc:729] T b0d7752eb0574dd1a86c55f68fa47b68 P 60502677e1b44304a598a89d364e31ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60502677e1b44304a598a89d364e31ee, State: Initialized, Role: FOLLOWER
I20260927 12:44:46.177393 17903 raft_consensus.cc:729] T b0d7752eb0574dd1a86c55f68fa47b68 P 892c00d9cd8a47b2b1121ca2b59df6ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 892c00d9cd8a47b2b1121ca2b59df6ba, State: Initialized, Role: FOLLOWER
I20260927 12:44:46.177874 17903 consensus_queue.cc:260] T b0d7752eb0574dd1a86c55f68fa47b68 P 892c00d9cd8a47b2b1121ca2b59df6ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "892c00d9cd8a47b2b1121ca2b59df6ba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33001 } } peers { permanent_uuid: "60502677e1b44304a598a89d364e31ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40551 } } peers { permanent_uuid: "a5187e9123af4d1eaadab092a3f72576" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33507 } }
I20260927 12:44:46.177874 17902 consensus_queue.cc:260] T b0d7752eb0574dd1a86c55f68fa47b68 P 60502677e1b44304a598a89d364e31ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "892c00d9cd8a47b2b1121ca2b59df6ba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33001 } } peers { permanent_uuid: "60502677e1b44304a598a89d364e31ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40551 } } peers { permanent_uuid: "a5187e9123af4d1eaadab092a3f72576" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33507 } }
I20260927 12:44:46.178202 17901 ts_tablet_manager.cc:1428] T b0d7752eb0574dd1a86c55f68fa47b68 P a5187e9123af4d1eaadab092a3f72576: Time spent starting tablet: real 0.006s user 0.001s sys 0.005s
I20260927 12:44:46.178576 17750 heartbeater.cc:503] Master 127.0.98.62:37047 was elected leader, sending a full tablet report...
I20260927 12:44:46.178833 17903 ts_tablet_manager.cc:1428] T b0d7752eb0574dd1a86c55f68fa47b68 P 892c00d9cd8a47b2b1121ca2b59df6ba: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 12:44:46.178833 17902 ts_tablet_manager.cc:1428] T b0d7752eb0574dd1a86c55f68fa47b68 P 60502677e1b44304a598a89d364e31ee: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 12:44:46.179186 17884 heartbeater.cc:503] Master 127.0.98.62:37047 was elected leader, sending a full tablet report...
I20260927 12:44:46.179841 17616 heartbeater.cc:503] Master 127.0.98.62:37047 was elected leader, sending a full tablet report...
I20260927 12:44:46.331370 17909 raft_consensus.cc:482] T b0d7752eb0574dd1a86c55f68fa47b68 P 60502677e1b44304a598a89d364e31ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:44:46.331614 17909 raft_consensus.cc:504] T b0d7752eb0574dd1a86c55f68fa47b68 P 60502677e1b44304a598a89d364e31ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "892c00d9cd8a47b2b1121ca2b59df6ba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33001 } } peers { permanent_uuid: "60502677e1b44304a598a89d364e31ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40551 } } peers { permanent_uuid: "a5187e9123af4d1eaadab092a3f72576" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33507 } }
I20260927 12:44:46.332942 17909 leader_election.cc:288] T b0d7752eb0574dd1a86c55f68fa47b68 P 60502677e1b44304a598a89d364e31ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 892c00d9cd8a47b2b1121ca2b59df6ba (127.0.98.3:33001), a5187e9123af4d1eaadab092a3f72576 (127.0.98.1:33507)
I20260927 12:44:46.337381 17900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:46.333389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49723 (local address 127.0.98.1:33507)
0927 12:44:46.333506 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:46.333511 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:46.333781 (+ 270us) server_negotiation.cc:407] Connection header received
0927 12:44:46.333900 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:46.333905 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:46.333956 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:46.334025 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:44:46.334376 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:46.335162 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:46.336422 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:46.336819 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:46.336936 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:46.337000 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:46.337184 (+ 184us) server_negotiation.cc:299] Negotiation successful
0927 12:44:46.337260 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 12:44:46.337397 17910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:46.333297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33507 (local address 127.0.98.2:49723)
0927 12:44:46.333687 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:46.333699 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 12:44:46.333801 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:44:46.334054 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:44:46.334056 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:44:46.334067 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:44:46.334244 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:46.334253 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:46.335332 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:46.335339 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:46.336288 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:46.336297 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:46.336939 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:46.336943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:46.337041 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:46.337083 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 12:44:46.337170 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 12:44:46.337237 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"mutex_wait_us":45,"thread_start_us":129,"threads_started":1}
I20260927 12:44:46.337409 17898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:46.333319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46117 (local address 127.0.98.3:33001)
0927 12:44:46.333474 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:46.333479 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:46.333498 (+ 19us) server_negotiation.cc:407] Connection header received
0927 12:44:46.333772 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:46.333776 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:46.333855 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:46.333942 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:44:46.334362 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:46.335162 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:46.336418 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:46.336833 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:46.336955 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:46.337020 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:46.337222 (+ 202us) server_negotiation.cc:299] Negotiation successful
0927 12:44:46.337293 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 12:44:46.337409 17752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:46.333187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33001 (local address 127.0.98.2:46117)
0927 12:44:46.333412 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:46.333429 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 12:44:46.333545 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:44:46.333958 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:44:46.333960 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:44:46.333973 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:44:46.334175 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:46.334182 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:46.335287 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:46.335293 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:46.336258 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:46.336267 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:46.336955 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:46.336958 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:46.337095 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:46.337111 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 12:44:46.337204 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 12:44:46.337286 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":84}
I20260927 12:44:46.338119 17838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d7752eb0574dd1a86c55f68fa47b68" candidate_uuid: "60502677e1b44304a598a89d364e31ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "892c00d9cd8a47b2b1121ca2b59df6ba" is_pre_election: true
I20260927 12:44:46.338337 17570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d7752eb0574dd1a86c55f68fa47b68" candidate_uuid: "60502677e1b44304a598a89d364e31ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5187e9123af4d1eaadab092a3f72576" is_pre_election: true
I20260927 12:44:46.338460 17838 raft_consensus.cc:2454] T b0d7752eb0574dd1a86c55f68fa47b68 P 892c00d9cd8a47b2b1121ca2b59df6ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60502677e1b44304a598a89d364e31ee in term 0.
I20260927 12:44:46.338608 17570 raft_consensus.cc:2454] T b0d7752eb0574dd1a86c55f68fa47b68 P a5187e9123af4d1eaadab092a3f72576 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60502677e1b44304a598a89d364e31ee in term 0.
I20260927 12:44:46.339016 17639 leader_election.cc:302] T b0d7752eb0574dd1a86c55f68fa47b68 P 60502677e1b44304a598a89d364e31ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 60502677e1b44304a598a89d364e31ee, 892c00d9cd8a47b2b1121ca2b59df6ba, a5187e9123af4d1eaadab092a3f72576; no voters:
I20260927 12:44:46.339320 17909 raft_consensus.cc:2781] T b0d7752eb0574dd1a86c55f68fa47b68 P 60502677e1b44304a598a89d364e31ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:44:46.339427 17909 raft_consensus.cc:482] T b0d7752eb0574dd1a86c55f68fa47b68 P 60502677e1b44304a598a89d364e31ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:44:46.339488 17909 raft_consensus.cc:3037] T b0d7752eb0574dd1a86c55f68fa47b68 P 60502677e1b44304a598a89d364e31ee [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:44:46.340484 17909 raft_consensus.cc:504] T b0d7752eb0574dd1a86c55f68fa47b68 P 60502677e1b44304a598a89d364e31ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "892c00d9cd8a47b2b1121ca2b59df6ba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33001 } } peers { permanent_uuid: "60502677e1b44304a598a89d364e31ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40551 } } peers { permanent_uuid: "a5187e9123af4d1eaadab092a3f72576" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33507 } }
W20260927 12:44:46.340812 17885 tablet.cc:2369] T b0d7752eb0574dd1a86c55f68fa47b68 P 892c00d9cd8a47b2b1121ca2b59df6ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:44:46.340900 17909 leader_election.cc:288] T b0d7752eb0574dd1a86c55f68fa47b68 P 60502677e1b44304a598a89d364e31ee [CANDIDATE]: Term 1 election: Requested vote from peers 892c00d9cd8a47b2b1121ca2b59df6ba (127.0.98.3:33001), a5187e9123af4d1eaadab092a3f72576 (127.0.98.1:33507)
I20260927 12:44:46.341228 17838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d7752eb0574dd1a86c55f68fa47b68" candidate_uuid: "60502677e1b44304a598a89d364e31ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "892c00d9cd8a47b2b1121ca2b59df6ba"
I20260927 12:44:46.341363 17838 raft_consensus.cc:3037] T b0d7752eb0574dd1a86c55f68fa47b68 P 892c00d9cd8a47b2b1121ca2b59df6ba [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:44:46.341689 17570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0d7752eb0574dd1a86c55f68fa47b68" candidate_uuid: "60502677e1b44304a598a89d364e31ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5187e9123af4d1eaadab092a3f72576"
I20260927 12:44:46.341832 17570 raft_consensus.cc:3037] T b0d7752eb0574dd1a86c55f68fa47b68 P a5187e9123af4d1eaadab092a3f72576 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:44:46.342483 17838 raft_consensus.cc:2454] T b0d7752eb0574dd1a86c55f68fa47b68 P 892c00d9cd8a47b2b1121ca2b59df6ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60502677e1b44304a598a89d364e31ee in term 1.
I20260927 12:44:46.342801 17639 leader_election.cc:302] T b0d7752eb0574dd1a86c55f68fa47b68 P 60502677e1b44304a598a89d364e31ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60502677e1b44304a598a89d364e31ee, 892c00d9cd8a47b2b1121ca2b59df6ba; no voters:
I20260927 12:44:46.342953 17909 raft_consensus.cc:2781] T b0d7752eb0574dd1a86c55f68fa47b68 P 60502677e1b44304a598a89d364e31ee [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:44:46.343137 17909 raft_consensus.cc:686] T b0d7752eb0574dd1a86c55f68fa47b68 P 60502677e1b44304a598a89d364e31ee [term 1 LEADER]: Becoming Leader. State: Replica: 60502677e1b44304a598a89d364e31ee, State: Running, Role: LEADER
I20260927 12:44:46.343221 17570 raft_consensus.cc:2454] T b0d7752eb0574dd1a86c55f68fa47b68 P a5187e9123af4d1eaadab092a3f72576 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60502677e1b44304a598a89d364e31ee in term 1.
I20260927 12:44:46.343480 17909 consensus_queue.cc:237] T b0d7752eb0574dd1a86c55f68fa47b68 P 60502677e1b44304a598a89d364e31ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "892c00d9cd8a47b2b1121ca2b59df6ba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33001 } } peers { permanent_uuid: "60502677e1b44304a598a89d364e31ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40551 } } peers { permanent_uuid: "a5187e9123af4d1eaadab092a3f72576" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33507 } }
I20260927 12:44:46.346796 17424 catalog_manager.cc:5495] T b0d7752eb0574dd1a86c55f68fa47b68 P 60502677e1b44304a598a89d364e31ee reported cstate change: term changed from 0 to 1, leader changed from <none> to 60502677e1b44304a598a89d364e31ee (127.0.98.2). New cstate: current_term: 1 leader_uuid: "60502677e1b44304a598a89d364e31ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "892c00d9cd8a47b2b1121ca2b59df6ba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60502677e1b44304a598a89d364e31ee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5187e9123af4d1eaadab092a3f72576" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33507 } health_report { overall_health: UNKNOWN } } }
W20260927 12:44:46.371325 17751 tablet.cc:2369] T b0d7752eb0574dd1a86c55f68fa47b68 P 60502677e1b44304a598a89d364e31ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:44:46.389567 17896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:46.385923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59830 (local address 127.0.98.2:40551)
0927 12:44:46.386058 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:46.386062 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:46.386077 (+ 15us) server_negotiation.cc:407] Connection header received
0927 12:44:46.386217 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:46.386220 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:46.386279 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:46.386374 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:44:46.386567 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:46.387123 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:46.388011 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:46.388274 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:46.388364 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:46.388626 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:44:46.389075 (+ 449us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:44:46.389210 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:46.389294 (+ 84us) server_negotiation.cc:299] Negotiation successful
0927 12:44:46.389346 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 12:44:46.396188 17838 raft_consensus.cc:1261] T b0d7752eb0574dd1a86c55f68fa47b68 P 892c00d9cd8a47b2b1121ca2b59df6ba [term 1 FOLLOWER]: Refusing update from remote peer 60502677e1b44304a598a89d364e31ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:44:46.396188 17570 raft_consensus.cc:1261] T b0d7752eb0574dd1a86c55f68fa47b68 P a5187e9123af4d1eaadab092a3f72576 [term 1 FOLLOWER]: Refusing update from remote peer 60502677e1b44304a598a89d364e31ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:44:46.397037 17911 consensus_queue.cc:1035] T b0d7752eb0574dd1a86c55f68fa47b68 P 60502677e1b44304a598a89d364e31ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "892c00d9cd8a47b2b1121ca2b59df6ba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:44:46.397259 17909 consensus_queue.cc:1035] T b0d7752eb0574dd1a86c55f68fa47b68 P 60502677e1b44304a598a89d364e31ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5187e9123af4d1eaadab092a3f72576" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:44:46.404508 17914 mvcc.cc:205] Tried to move back new op lower bound from 7333941601870467072 to 7333941601665753088. Current Snapshot: MvccSnapshot[applied={T|T < 7333941601870467072}]
I20260927 12:44:46.405879 17915 mvcc.cc:205] Tried to move back new op lower bound from 7333941601870467072 to 7333941601665753088. Current Snapshot: MvccSnapshot[applied={T|T < 7333941601870467072}]
I20260927 12:44:46.406251 17916 mvcc.cc:205] Tried to move back new op lower bound from 7333941601870467072 to 7333941601665753088. Current Snapshot: MvccSnapshot[applied={T|T < 7333941601870467072}]
I20260927 12:44:46.413786 17423 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:44:46.413970 17423 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:44:46.416294 17423 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 798b2baecc6c4090bff49464ca14c5db: Sending DeleteTablet for 3 replicas of tablet b0d7752eb0574dd1a86c55f68fa47b68
I20260927 12:44:46.417043 17818 tablet_service.cc:1514] Processing DeleteTablet for tablet b0d7752eb0574dd1a86c55f68fa47b68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:44:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58120
I20260927 12:44:46.417119 17550 tablet_service.cc:1514] Processing DeleteTablet for tablet b0d7752eb0574dd1a86c55f68fa47b68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:44:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51810
I20260927 12:44:46.417338 17684 tablet_service.cc:1514] Processing DeleteTablet for tablet b0d7752eb0574dd1a86c55f68fa47b68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:44:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59826
I20260927 12:44:46.417737 17924 tablet_replica.cc:330] T b0d7752eb0574dd1a86c55f68fa47b68 P a5187e9123af4d1eaadab092a3f72576: stopping tablet replica
I20260927 12:44:46.417789 17925 tablet_replica.cc:330] T b0d7752eb0574dd1a86c55f68fa47b68 P 60502677e1b44304a598a89d364e31ee: stopping tablet replica
I20260927 12:44:46.418092 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 17487
I20260927 12:44:46.418128 17925 raft_consensus.cc:2229] T b0d7752eb0574dd1a86c55f68fa47b68 P 60502677e1b44304a598a89d364e31ee [term 1 LEADER]: Raft consensus shutting down.
I20260927 12:44:46.418279 17924 raft_consensus.cc:2229] T b0d7752eb0574dd1a86c55f68fa47b68 P a5187e9123af4d1eaadab092a3f72576 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:44:46.419405 17925 raft_consensus.cc:2258] T b0d7752eb0574dd1a86c55f68fa47b68 P 60502677e1b44304a598a89d364e31ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:44:46.420921 17925 ts_tablet_manager.cc:1905] T b0d7752eb0574dd1a86c55f68fa47b68 P 60502677e1b44304a598a89d364e31ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:44:46.421944 17923 tablet_replica.cc:330] T b0d7752eb0574dd1a86c55f68fa47b68 P 892c00d9cd8a47b2b1121ca2b59df6ba: stopping tablet replica
I20260927 12:44:46.422312 17923 raft_consensus.cc:2229] T b0d7752eb0574dd1a86c55f68fa47b68 P 892c00d9cd8a47b2b1121ca2b59df6ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:44:46.422689 17923 raft_consensus.cc:2258] T b0d7752eb0574dd1a86c55f68fa47b68 P 892c00d9cd8a47b2b1121ca2b59df6ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:44:46.424144 17923 ts_tablet_manager.cc:1905] T b0d7752eb0574dd1a86c55f68fa47b68 P 892c00d9cd8a47b2b1121ca2b59df6ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:44:46.424156 17925 ts_tablet_manager.cc:1918] T b0d7752eb0574dd1a86c55f68fa47b68 P 60502677e1b44304a598a89d364e31ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:44:46.424248 17925 log.cc:1196] T b0d7752eb0574dd1a86c55f68fa47b68 P 60502677e1b44304a598a89d364e31ee: Deleting WAL directory at /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/minicluster-data/ts-1/wal/wals/b0d7752eb0574dd1a86c55f68fa47b68
I20260927 12:44:46.424559 17925 ts_tablet_manager.cc:1939] T b0d7752eb0574dd1a86c55f68fa47b68 P 60502677e1b44304a598a89d364e31ee: Deleting consensus metadata
I20260927 12:44:46.425971 17410 catalog_manager.cc:4841] TS 60502677e1b44304a598a89d364e31ee (127.0.98.2:40551): tablet b0d7752eb0574dd1a86c55f68fa47b68 (table test-table [id=2ad7ebbda50d465daffd7901ac51996d]) successfully deleted
W20260927 12:44:46.426177 17409 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33507 (error 108)
I20260927 12:44:46.427094 17923 ts_tablet_manager.cc:1918] T b0d7752eb0574dd1a86c55f68fa47b68 P 892c00d9cd8a47b2b1121ca2b59df6ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:44:46.427182 17923 log.cc:1196] T b0d7752eb0574dd1a86c55f68fa47b68 P 892c00d9cd8a47b2b1121ca2b59df6ba: Deleting WAL directory at /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790512840836255-392-0/minicluster-data/ts-2/wal/wals/b0d7752eb0574dd1a86c55f68fa47b68
I20260927 12:44:46.427220 17899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:46.426882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33507 (local address 127.0.0.1:51818)
0927 12:44:46.427039 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:46.427118 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 12:44:46.427245 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 17621
I20260927 12:44:46.427506 17923 ts_tablet_manager.cc:1939] T b0d7752eb0574dd1a86c55f68fa47b68 P 892c00d9cd8a47b2b1121ca2b59df6ba: Deleting consensus metadata
W20260927 12:44:46.428663 17409 catalog_manager.cc:4584] TS a5187e9123af4d1eaadab092a3f72576 (127.0.98.1:33507): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b0d7752eb0574dd1a86c55f68fa47b68: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33507: connect: Connection refused (error 111)
I20260927 12:44:46.428949 17411 catalog_manager.cc:4841] TS 892c00d9cd8a47b2b1121ca2b59df6ba (127.0.98.3:33001): tablet b0d7752eb0574dd1a86c55f68fa47b68 (table test-table [id=2ad7ebbda50d465daffd7901ac51996d]) successfully deleted
I20260927 12:44:46.433761 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 17755
I20260927 12:44:46.439131 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 17393
2026-09-27T12:44:46Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2424 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-27T12:44:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:44:46Z Disabled control of system clock
I20260927 12:44:46.469035 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512840836255-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512840836255-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512840836255-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43091
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:43249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43091
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 12:44:46.578279 17930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:44:46.578536 17930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:44:46.578572 17930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:44:46.582470 17930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:44:46.582538 17930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:44:46.582553 17930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:44:46.582571 17930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:44:46.582587 17930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:44:46.587798 17930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512840836255-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512840836255-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43091
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512840836255-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:44:46.588900 17930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:44:46.590302 17930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 12:44:46.598910 17930 server_base.cc:1034] running on GCE node
W20260927 12:44:46.598860 17935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:44:46.598866 17936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:44:46.598860 17938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:44:46.599625 17930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:44:46.600714 17930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:44:46.601899 17930 hybrid_clock.cc:648] HybridClock initialized: now 1790513086601884 us; error 27 us; skew 500 ppm
I20260927 12:44:46.603822 17930 webserver.cc:460] Webserver started at http://127.0.98.62:44973/ using document root <none> and password file <none>
I20260927 12:44:46.604460 17930 fs_manager.cc:359] Metadata directory not provided
I20260927 12:44:46.604548 17930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:44:46.604780 17930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:44:46.607542 17930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512840836255-392-0/minicluster-data/master-0/data/instance:
uuid: "6852b84b02b84ea5a3a77204476ae7a5"
format_stamp: "Formatted at 2026-09-27 12:44:46 on dist-test-slave-bnc1"
server_key: "1693bc840b2528d4799b3393968b6bd9"
server_key_iv: "c7292c30f0a2dba9ae9f974f06b6d5a4"
server_key_version: "encryptionkey@0"
I20260927 12:44:46.608331 17930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512840836255-392-0/minicluster-data/master-0/wal/instance:
uuid: "6852b84b02b84ea5a3a77204476ae7a5"
format_stamp: "Formatted at 2026-09-27 12:44:46 on dist-test-slave-bnc1"
server_key: "1693bc840b2528d4799b3393968b6bd9"
server_key_iv: "c7292c30f0a2dba9ae9f974f06b6d5a4"
server_key_version: "encryptionkey@0"
I20260927 12:44:46.613318 17930 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260927 12:44:46.617357 17944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:44:46.618827 17930 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260927 12:44:46.618963 17930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512840836255-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512840836255-392-0/minicluster-data/master-0/wal
uuid: "6852b84b02b84ea5a3a77204476ae7a5"
format_stamp: "Formatted at 2026-09-27 12:44:46 on dist-test-slave-bnc1"
server_key: "1693bc840b2528d4799b3393968b6bd9"
server_key_iv: "c7292c30f0a2dba9ae9f974f06b6d5a4"
server_key_version: "encryptionkey@0"
I20260927 12:44:46.619083 17930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512840836255-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512840836255-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512840836255-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:44:46.646842 17930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:44:46.647644 17930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:44:46.647886 17930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:44:46.655512 17930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43091
I20260927 12:44:46.655529 17996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43091 every 8 connection(s)
I20260927 12:44:46.656553 17930 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:44:46.656884 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 17930
I20260927 12:44:46.657007 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512840836255-392-0/minicluster-data/master-0/wal/instance
I20260927 12:44:46.657354 392 external_mini_cluster.cc:1428] Setting key 3cb996ae210f02fe53b119b9bca141f3
I20260927 12:44:46.660100 17997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:44:46.666741 17997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6852b84b02b84ea5a3a77204476ae7a5: Bootstrap starting.
I20260927 12:44:46.666904 18000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:46.658899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39812 (local address 127.0.98.62:43091)
0927 12:44:46.659507 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:46.659524 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:46.659557 (+ 33us) server_negotiation.cc:407] Connection header received
0927 12:44:46.660178 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:46.660193 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:46.660632 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:46.660925 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:44:46.661307 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:46.662349 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:46.663607 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:46.664057 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:46.664173 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:46.665097 (+ 924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:44:46.665102 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:44:46.665114 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:44:46.665513 (+ 399us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:44:46.665613 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:44:46.665646 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:44:46.665723 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:46.665894 (+ 171us) server_negotiation.cc:299] Negotiation successful
0927 12:44:46.666178 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":131,"threads_started":1}
I20260927 12:44:46.669253 17997 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6852b84b02b84ea5a3a77204476ae7a5: Neither blocks nor log segments found. Creating new log.
I20260927 12:44:46.670061 17997 log.cc:824] T 00000000000000000000000000000000 P 6852b84b02b84ea5a3a77204476ae7a5: Log is configured to *not* fsync() on all Append() calls
I20260927 12:44:46.671893 17997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6852b84b02b84ea5a3a77204476ae7a5: No bootstrap required, opened a new log
I20260927 12:44:46.675436 17997 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6852b84b02b84ea5a3a77204476ae7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6852b84b02b84ea5a3a77204476ae7a5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43091 } }
I20260927 12:44:46.675691 17997 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6852b84b02b84ea5a3a77204476ae7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:44:46.675776 17997 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6852b84b02b84ea5a3a77204476ae7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6852b84b02b84ea5a3a77204476ae7a5, State: Initialized, Role: FOLLOWER
I20260927 12:44:46.676200 17997 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6852b84b02b84ea5a3a77204476ae7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6852b84b02b84ea5a3a77204476ae7a5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43091 } }
I20260927 12:44:46.676378 17997 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6852b84b02b84ea5a3a77204476ae7a5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:44:46.676456 17997 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6852b84b02b84ea5a3a77204476ae7a5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:44:46.676584 17997 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6852b84b02b84ea5a3a77204476ae7a5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:44:46.677505 17997 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6852b84b02b84ea5a3a77204476ae7a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6852b84b02b84ea5a3a77204476ae7a5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43091 } }
I20260927 12:44:46.677904 17997 leader_election.cc:302] T 00000000000000000000000000000000 P 6852b84b02b84ea5a3a77204476ae7a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6852b84b02b84ea5a3a77204476ae7a5; no voters:
I20260927 12:44:46.678231 17997 leader_election.cc:288] T 00000000000000000000000000000000 P 6852b84b02b84ea5a3a77204476ae7a5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:44:46.678428 18002 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6852b84b02b84ea5a3a77204476ae7a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:44:46.678646 18002 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6852b84b02b84ea5a3a77204476ae7a5 [term 1 LEADER]: Becoming Leader. State: Replica: 6852b84b02b84ea5a3a77204476ae7a5, State: Running, Role: LEADER
I20260927 12:44:46.678946 18002 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6852b84b02b84ea5a3a77204476ae7a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6852b84b02b84ea5a3a77204476ae7a5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43091 } }
I20260927 12:44:46.679497 17997 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6852b84b02b84ea5a3a77204476ae7a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:44:46.680591 18003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6852b84b02b84ea5a3a77204476ae7a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6852b84b02b84ea5a3a77204476ae7a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6852b84b02b84ea5a3a77204476ae7a5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43091 } } }
I20260927 12:44:46.680742 18003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6852b84b02b84ea5a3a77204476ae7a5 [sys.catalog]: This master's current role is: LEADER
I20260927 12:44:46.681043 18004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6852b84b02b84ea5a3a77204476ae7a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6852b84b02b84ea5a3a77204476ae7a5. Latest consensus state: current_term: 1 leader_uuid: "6852b84b02b84ea5a3a77204476ae7a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6852b84b02b84ea5a3a77204476ae7a5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43091 } } }
I20260927 12:44:46.681135 18011 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:44:46.681149 18004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6852b84b02b84ea5a3a77204476ae7a5 [sys.catalog]: This master's current role is: LEADER
I20260927 12:44:46.684418 18011 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:44:46.689877 18011 catalog_manager.cc:1347] Generated new cluster ID: 871488c393824eabbc26092049b86d19
I20260927 12:44:46.689946 18011 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:44:46.700475 18011 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:44:46.701758 18011 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:44:46.707091 18011 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6852b84b02b84ea5a3a77204476ae7a5: Generated new TSK 0
I20260927 12:44:46.707834 18011 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:44:46.718647 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512840836255-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512840836255-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512840836255-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43091
--builtin_ntp_servers=127.0.98.20:43249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 12:44:46.831843 18021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:44:46.832109 18021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:44:46.832149 18021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:44:46.836196 18021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:44:46.836257 18021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:44:46.836313 18021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:44:46.840940 18021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512840836255-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512840836255-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512840836255-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:44:46.841995 18021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:44:46.843302 18021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 12:44:46.853289 18021 server_base.cc:1034] running on GCE node
W20260927 12:44:46.853228 18027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:44:46.853231 18026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:44:46.853271 18029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:44:46.853947 18021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:44:46.854700 18021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:44:46.855914 18021 hybrid_clock.cc:648] HybridClock initialized: now 1790513086855886 us; error 39 us; skew 500 ppm
I20260927 12:44:46.858073 18021 webserver.cc:460] Webserver started at http://127.0.98.1:34259/ using document root <none> and password file <none>
I20260927 12:44:46.858682 18021 fs_manager.cc:359] Metadata directory not provided
I20260927 12:44:46.858779 18021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:44:46.859018 18021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:44:46.861127 18021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512840836255-392-0/minicluster-data/ts-0/data/instance:
uuid: "6cf2886496cc41f380499905bfd0cdbc"
format_stamp: "Formatted at 2026-09-27 12:44:46 on dist-test-slave-bnc1"
server_key: "a90a41848c34115b4796380962e47591"
server_key_iv: "d116d5729e1d52b5e6b579192d46e24c"
server_key_version: "encryptionkey@0"
I20260927 12:44:46.861779 18021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512840836255-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6cf2886496cc41f380499905bfd0cdbc"
format_stamp: "Formatted at 2026-09-27 12:44:46 on dist-test-slave-bnc1"
server_key: "a90a41848c34115b4796380962e47591"
server_key_iv: "d116d5729e1d52b5e6b579192d46e24c"
server_key_version: "encryptionkey@0"
I20260927 12:44:46.865749 18021 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 12:44:46.868327 18035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:44:46.869536 18021 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:44:46.869705 18021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512840836255-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512840836255-392-0/minicluster-data/ts-0/wal
uuid: "6cf2886496cc41f380499905bfd0cdbc"
format_stamp: "Formatted at 2026-09-27 12:44:46 on dist-test-slave-bnc1"
server_key: "a90a41848c34115b4796380962e47591"
server_key_iv: "d116d5729e1d52b5e6b579192d46e24c"
server_key_version: "encryptionkey@0"
I20260927 12:44:46.869827 18021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512840836255-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512840836255-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512840836255-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:44:46.893700 18021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:44:46.894462 18021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:44:46.894662 18021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:44:46.895324 18021 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:44:46.896284 18021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:44:46.896355 18021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:44:46.896428 18021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:44:46.896471 18021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:44:46.906069 18021 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45125
I20260927 12:44:46.906097 18148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45125 every 8 connection(s)
I20260927 12:44:46.906970 18021 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:44:46.915375 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 18021
I20260927 12:44:46.915515 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512840836255-392-0/minicluster-data/ts-0/wal/instance
I20260927 12:44:46.915877 392 external_mini_cluster.cc:1428] Setting key 83206baea61e3b716dbc122348ce5fbb
I20260927 12:44:46.916112 18000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:46.908788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52111 (local address 127.0.98.62:43091)
0927 12:44:46.908923 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:46.908927 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:46.909973 (+ 1046us) server_negotiation.cc:407] Connection header received
0927 12:44:46.911337 (+ 1364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:46.911340 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:46.911390 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:46.911472 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:44:46.912359 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:46.912900 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:46.914117 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:46.914339 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:46.914422 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:46.915582 (+ 1160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:44:46.915586 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:44:46.915589 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:44:46.915603 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:44:46.915624 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:44:46.915641 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:44:46.915701 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:46.915890 (+ 189us) server_negotiation.cc:299] Negotiation successful
0927 12:44:46.915988 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 12:44:46.917140 18151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:46.909181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43091 (local address 127.0.98.1:52111)
0927 12:44:46.909820 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:46.909860 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 12:44:46.911093 (+ 1233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:44:46.911633 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:44:46.911639 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:44:46.911724 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:44:46.912199 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:46.912209 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:46.913004 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:46.913006 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:46.914005 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:46.914014 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:46.914474 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:46.914476 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:46.914612 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:46.915246 (+ 634us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 12:44:46.915251 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:44:46.915309 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 12:44:46.915395 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:44:46.915732 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:44:46.915742 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:44:46.915749 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 12:44:46.915883 (+ 134us) client_negotiation.cc:240] Negotiation successful
0927 12:44:46.916159 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":102,"threads_started":1}
I20260927 12:44:46.917958 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512840836255-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512840836255-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512840836255-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43091
--builtin_ntp_servers=127.0.98.20:43249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 12:44:46.918390 18149 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43091
I20260927 12:44:46.918610 18149 heartbeater.cc:464] Registering TS with master...
I20260927 12:44:46.919178 18149 heartbeater.cc:511] Master 127.0.98.62:43091 requested a full tablet report, sending...
I20260927 12:44:46.920670 17961 ts_manager.cc:194] Registered new tserver with Master: 6cf2886496cc41f380499905bfd0cdbc (127.0.98.1:45125)
I20260927 12:44:46.921972 17961 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:52111
W20260927 12:44:47.030122 18152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:44:47.030409 18152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:44:47.030498 18152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:44:47.035001 18152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:44:47.035068 18152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:44:47.035183 18152 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:44:47.040405 18152 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512840836255-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512840836255-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512840836255-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:44:47.041554 18152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:44:47.043109 18152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:44:47.053601 18160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:44:47.053601 18158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:44:47.053715 18157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:44:47.053951 18152 server_base.cc:1034] running on GCE node
I20260927 12:44:47.054391 18152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:44:47.055116 18152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:44:47.056306 18152 hybrid_clock.cc:648] HybridClock initialized: now 1790513087056286 us; error 35 us; skew 500 ppm
I20260927 12:44:47.058472 18152 webserver.cc:460] Webserver started at http://127.0.98.2:39985/ using document root <none> and password file <none>
I20260927 12:44:47.059115 18152 fs_manager.cc:359] Metadata directory not provided
I20260927 12:44:47.059217 18152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:44:47.059489 18152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:44:47.061934 18152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512840836255-392-0/minicluster-data/ts-1/data/instance:
uuid: "d7e902c31074413b9dc80cb909e16de0"
format_stamp: "Formatted at 2026-09-27 12:44:47 on dist-test-slave-bnc1"
server_key: "21c6c49c500046b00983c40207d906ce"
server_key_iv: "5b6820334a6c9642ca3424b09eafabd0"
server_key_version: "encryptionkey@0"
I20260927 12:44:47.062603 18152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512840836255-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d7e902c31074413b9dc80cb909e16de0"
format_stamp: "Formatted at 2026-09-27 12:44:47 on dist-test-slave-bnc1"
server_key: "21c6c49c500046b00983c40207d906ce"
server_key_iv: "5b6820334a6c9642ca3424b09eafabd0"
server_key_version: "encryptionkey@0"
I20260927 12:44:47.067589 18152 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260927 12:44:47.070418 18166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:44:47.071735 18152 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 12:44:47.071875 18152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512840836255-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512840836255-392-0/minicluster-data/ts-1/wal
uuid: "d7e902c31074413b9dc80cb909e16de0"
format_stamp: "Formatted at 2026-09-27 12:44:47 on dist-test-slave-bnc1"
server_key: "21c6c49c500046b00983c40207d906ce"
server_key_iv: "5b6820334a6c9642ca3424b09eafabd0"
server_key_version: "encryptionkey@0"
I20260927 12:44:47.071991 18152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512840836255-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512840836255-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512840836255-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:44:47.086367 18152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:44:47.087095 18152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:44:47.087302 18152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:44:47.087904 18152 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:44:47.088843 18152 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:44:47.088897 18152 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:44:47.088961 18152 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:44:47.088994 18152 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:44:47.098842 18152 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40297
I20260927 12:44:47.098867 18279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40297 every 8 connection(s)
I20260927 12:44:47.099805 18152 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:44:47.104562 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 18152
I20260927 12:44:47.104753 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512840836255-392-0/minicluster-data/ts-1/wal/instance
I20260927 12:44:47.105183 392 external_mini_cluster.cc:1428] Setting key 0beceeb67a2a6c9a23a9ee282df32ce4
I20260927 12:44:47.107803 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512840836255-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512840836255-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512840836255-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43091
--builtin_ntp_servers=127.0.98.20:43249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 12:44:47.111264 18000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:47.101793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40927 (local address 127.0.98.62:43091)
0927 12:44:47.101951 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:47.101954 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:47.103184 (+ 1230us) server_negotiation.cc:407] Connection header received
0927 12:44:47.105083 (+ 1899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:47.105086 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:47.105151 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:47.105234 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:44:47.106489 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:47.107110 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:47.109109 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:47.109328 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:47.109451 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:47.110713 (+ 1262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:44:47.110723 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:44:47.110726 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:44:47.110746 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:44:47.110766 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:44:47.110774 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:44:47.110852 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:47.111045 (+ 193us) server_negotiation.cc:299] Negotiation successful
0927 12:44:47.111117 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 12:44:47.112164 18282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:47.102214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43091 (local address 127.0.98.2:40927)
0927 12:44:47.102965 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:47.103031 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0927 12:44:47.104814 (+ 1783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:44:47.105454 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:44:47.105466 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:44:47.105618 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:44:47.106323 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:47.106345 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:47.107261 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:47.107265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:47.108951 (+ 1686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:47.108969 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:47.109500 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:47.109504 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:47.109750 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:47.110420 (+ 670us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 12:44:47.110422 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:44:47.110466 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 12:44:47.110538 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:44:47.110865 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:44:47.110876 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:44:47.110888 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 12:44:47.111025 (+ 137us) client_negotiation.cc:240] Negotiation successful
0927 12:44:47.111347 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":110,"threads_started":1}
I20260927 12:44:47.113420 18280 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43091
I20260927 12:44:47.113692 18280 heartbeater.cc:464] Registering TS with master...
I20260927 12:44:47.114259 18280 heartbeater.cc:511] Master 127.0.98.62:43091 requested a full tablet report, sending...
I20260927 12:44:47.115396 17961 ts_manager.cc:194] Registered new tserver with Master: d7e902c31074413b9dc80cb909e16de0 (127.0.98.2:40297)
I20260927 12:44:47.116402 17961 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:40927
W20260927 12:44:47.223724 18283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:44:47.224033 18283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:44:47.224073 18283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:44:47.228049 18283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:44:47.228109 18283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:44:47.228168 18283 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:44:47.232792 18283 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512840836255-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512840836255-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18283
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512840836255-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:44:47.233886 18283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:44:47.235293 18283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:44:47.244741 18289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:44:47.244886 18283 server_base.cc:1034] running on GCE node
W20260927 12:44:47.244757 18288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:44:47.244750 18291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:44:47.245432 18283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:44:47.246105 18283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:44:47.247293 18283 hybrid_clock.cc:648] HybridClock initialized: now 1790513087247276 us; error 29 us; skew 500 ppm
I20260927 12:44:47.249109 18283 webserver.cc:460] Webserver started at http://127.0.98.3:45367/ using document root <none> and password file <none>
I20260927 12:44:47.249744 18283 fs_manager.cc:359] Metadata directory not provided
I20260927 12:44:47.249835 18283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:44:47.250038 18283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:44:47.252015 18283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512840836255-392-0/minicluster-data/ts-2/data/instance:
uuid: "5c42b8c62b174737936e4dd6d9cba374"
format_stamp: "Formatted at 2026-09-27 12:44:47 on dist-test-slave-bnc1"
server_key: "b32ebe7a454e3df2770fd6d9c8406e6b"
server_key_iv: "e7d7b77aab469d6ad910f0fb130f657c"
server_key_version: "encryptionkey@0"
I20260927 12:44:47.252588 18283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512840836255-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5c42b8c62b174737936e4dd6d9cba374"
format_stamp: "Formatted at 2026-09-27 12:44:47 on dist-test-slave-bnc1"
server_key: "b32ebe7a454e3df2770fd6d9c8406e6b"
server_key_iv: "e7d7b77aab469d6ad910f0fb130f657c"
server_key_version: "encryptionkey@0"
I20260927 12:44:47.256229 18283 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 12:44:47.258673 18297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:44:47.259927 18283 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 12:44:47.260069 18283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512840836255-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512840836255-392-0/minicluster-data/ts-2/wal
uuid: "5c42b8c62b174737936e4dd6d9cba374"
format_stamp: "Formatted at 2026-09-27 12:44:47 on dist-test-slave-bnc1"
server_key: "b32ebe7a454e3df2770fd6d9c8406e6b"
server_key_iv: "e7d7b77aab469d6ad910f0fb130f657c"
server_key_version: "encryptionkey@0"
I20260927 12:44:47.260211 18283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512840836255-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512840836255-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512840836255-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:44:47.275234 18283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:44:47.276016 18283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:44:47.276223 18283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:44:47.276877 18283 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:44:47.277935 18283 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:44:47.278007 18283 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:44:47.278077 18283 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:44:47.278129 18283 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:44:47.286968 18283 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42161
I20260927 12:44:47.286986 18410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42161 every 8 connection(s)
I20260927 12:44:47.287910 18283 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:44:47.295356 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 18283
I20260927 12:44:47.295463 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512840836255-392-0/minicluster-data/ts-2/wal/instance
I20260927 12:44:47.295814 392 external_mini_cluster.cc:1428] Setting key 990494506f6417d85d25fcf3e26a4441
I20260927 12:44:47.297868 18000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:47.289790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57841 (local address 127.0.98.62:43091)
0927 12:44:47.289933 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:47.289937 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:47.291022 (+ 1085us) server_negotiation.cc:407] Connection header received
0927 12:44:47.292457 (+ 1435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:47.292459 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:47.292509 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:47.292590 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:44:47.293511 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:47.294077 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:47.295261 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:47.295574 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:47.295659 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:47.296930 (+ 1271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:44:47.296933 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:44:47.296936 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:44:47.296956 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:44:47.296987 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:44:47.296996 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:44:47.297089 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:47.297663 (+ 574us) server_negotiation.cc:299] Negotiation successful
0927 12:44:47.297724 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 12:44:47.298527 18413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:47.290252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43091 (local address 127.0.98.3:57841)
0927 12:44:47.290891 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:47.290933 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 12:44:47.292234 (+ 1301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:44:47.292783 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:44:47.292789 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:44:47.292877 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:44:47.293375 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:47.293388 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:47.294186 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:47.294188 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:47.295129 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:47.295143 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:47.295755 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:47.295760 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:47.295965 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:47.296633 (+ 668us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 12:44:47.296634 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:44:47.296679 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 12:44:47.296785 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:44:47.297103 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:44:47.297110 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:44:47.297115 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 12:44:47.297314 (+ 199us) client_negotiation.cc:240] Negotiation successful
0927 12:44:47.297559 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":111,"threads_started":1}
I20260927 12:44:47.299520 18411 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43091
I20260927 12:44:47.299770 18411 heartbeater.cc:464] Registering TS with master...
I20260927 12:44:47.300297 18411 heartbeater.cc:511] Master 127.0.98.62:43091 requested a full tablet report, sending...
I20260927 12:44:47.301470 17961 ts_manager.cc:194] Registered new tserver with Master: 5c42b8c62b174737936e4dd6d9cba374 (127.0.98.3:42161)
I20260927 12:44:47.302127 17961 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:57841
I20260927 12:44:47.310070 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 12:44:47.315635 18000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:47.312077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39820 (local address 127.0.98.62:43091)
0927 12:44:47.312212 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:47.312243 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:47.312364 (+ 121us) server_negotiation.cc:407] Connection header received
0927 12:44:47.312452 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:47.312454 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:47.312494 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:47.312569 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:44:47.312821 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:47.313337 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:47.314172 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:47.314414 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:47.314496 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:47.315245 (+ 749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:44:47.315248 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:44:47.315250 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:44:47.315263 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:44:47.315284 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:44:47.315293 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:44:47.315358 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:47.315451 (+ 93us) server_negotiation.cc:299] Negotiation successful
0927 12:44:47.315514 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 12:44:47.318506 17961 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39820:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 12:44:47.321518 17961 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:44:47.335963 18423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:47.330110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41216 (local address 127.0.98.1:45125)
0927 12:44:47.330451 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:47.330463 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:47.330697 (+ 234us) server_negotiation.cc:407] Connection header received
0927 12:44:47.330915 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:47.330921 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:47.331152 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:47.331329 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:44:47.331639 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:47.332873 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:47.334665 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:47.335291 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:47.335418 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:47.335624 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:47.335722 (+ 98us) server_negotiation.cc:299] Negotiation successful
0927 12:44:47.335816 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":114,"threads_started":1}
I20260927 12:44:47.335952 18422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:47.329993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45125 (local address 127.0.0.1:41216)
0927 12:44:47.330528 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:47.330582 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 12:44:47.330729 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:44:47.331313 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:44:47.331318 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:44:47.331342 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:44:47.331495 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:47.331502 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:47.332998 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:47.333001 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:47.334530 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:47.334541 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:47.335417 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:47.335421 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:47.335529 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:47.335564 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 12:44:47.335674 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 12:44:47.335793 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":104,"thread_start_us":73,"threads_started":1}
I20260927 12:44:47.336165 18426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:47.331083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40297 (local address 127.0.0.1:57080)
0927 12:44:47.331590 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:47.331605 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:44:47.331681 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:44:47.332393 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:44:47.332397 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:44:47.332415 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:44:47.332496 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:47.332501 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:47.333696 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:47.333699 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:47.334774 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:47.334781 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:47.335805 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:47.335809 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:47.335892 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:47.335914 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 12:44:47.335967 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 12:44:47.336034 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"mutex_wait_us":1,"thread_start_us":87,"threads_started":1}
I20260927 12:44:47.336653 18424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:47.330762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42161 (local address 127.0.0.1:38634)
0927 12:44:47.331013 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:47.331026 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:44:47.331115 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:44:47.332262 (+ 1147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:44:47.332265 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:44:47.332283 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:44:47.332386 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:47.332396 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:47.333587 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:47.333591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:47.334737 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:47.334747 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:47.336003 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:47.336007 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:47.336371 (+ 364us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:47.336400 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 12:44:47.336473 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 12:44:47.336534 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"mutex_wait_us":1,"thread_start_us":115,"threads_started":1}
I20260927 12:44:47.336933 18427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:47.331500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57080 (local address 127.0.98.2:40297)
0927 12:44:47.331939 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:47.331950 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:47.331972 (+ 22us) server_negotiation.cc:407] Connection header received
0927 12:44:47.332046 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:47.332051 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:47.332235 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:47.332624 (+ 389us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:44:47.332746 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:47.333568 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:47.334921 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:47.335653 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:47.336418 (+ 765us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:47.336606 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:47.336656 (+ 50us) server_negotiation.cc:299] Negotiation successful
0927 12:44:47.336778 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":119,"threads_started":1}
I20260927 12:44:47.336862 18425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:47.330954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38634 (local address 127.0.98.3:42161)
0927 12:44:47.331909 (+ 955us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:47.331917 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:47.331934 (+ 17us) server_negotiation.cc:407] Connection header received
0927 12:44:47.331991 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:47.331994 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:47.332132 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:47.332281 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:44:47.332482 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:47.333431 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:47.334862 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:47.335653 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:47.335773 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:47.335948 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:47.336505 (+ 557us) server_negotiation.cc:299] Negotiation successful
0927 12:44:47.336606 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":870,"thread_start_us":167,"threads_started":1}
I20260927 12:44:47.339277 18083 tablet_service.cc:1467] Processing CreateTablet for tablet ea296376f7a94e938bd4e144f6becfb1 (DEFAULT_TABLE table=test-table [id=f9eaa9ce655b4dd3bdc743bd6f61542f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:44:47.339640 18214 tablet_service.cc:1467] Processing CreateTablet for tablet ea296376f7a94e938bd4e144f6becfb1 (DEFAULT_TABLE table=test-table [id=f9eaa9ce655b4dd3bdc743bd6f61542f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:44:47.340329 18083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea296376f7a94e938bd4e144f6becfb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:44:47.340234 18345 tablet_service.cc:1467] Processing CreateTablet for tablet ea296376f7a94e938bd4e144f6becfb1 (DEFAULT_TABLE table=test-table [id=f9eaa9ce655b4dd3bdc743bd6f61542f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:44:47.340662 18214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea296376f7a94e938bd4e144f6becfb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:44:47.341183 18345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea296376f7a94e938bd4e144f6becfb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:44:47.347078 18428 tablet_bootstrap.cc:492] T ea296376f7a94e938bd4e144f6becfb1 P 6cf2886496cc41f380499905bfd0cdbc: Bootstrap starting.
I20260927 12:44:47.348229 18429 tablet_bootstrap.cc:492] T ea296376f7a94e938bd4e144f6becfb1 P d7e902c31074413b9dc80cb909e16de0: Bootstrap starting.
I20260927 12:44:47.349221 18428 tablet_bootstrap.cc:654] T ea296376f7a94e938bd4e144f6becfb1 P 6cf2886496cc41f380499905bfd0cdbc: Neither blocks nor log segments found. Creating new log.
I20260927 12:44:47.349905 18430 tablet_bootstrap.cc:492] T ea296376f7a94e938bd4e144f6becfb1 P 5c42b8c62b174737936e4dd6d9cba374: Bootstrap starting.
I20260927 12:44:47.350023 18429 tablet_bootstrap.cc:654] T ea296376f7a94e938bd4e144f6becfb1 P d7e902c31074413b9dc80cb909e16de0: Neither blocks nor log segments found. Creating new log.
I20260927 12:44:47.350116 18428 log.cc:824] T ea296376f7a94e938bd4e144f6becfb1 P 6cf2886496cc41f380499905bfd0cdbc: Log is configured to *not* fsync() on all Append() calls
I20260927 12:44:47.350854 18429 log.cc:824] T ea296376f7a94e938bd4e144f6becfb1 P d7e902c31074413b9dc80cb909e16de0: Log is configured to *not* fsync() on all Append() calls
I20260927 12:44:47.351948 18430 tablet_bootstrap.cc:654] T ea296376f7a94e938bd4e144f6becfb1 P 5c42b8c62b174737936e4dd6d9cba374: Neither blocks nor log segments found. Creating new log.
I20260927 12:44:47.352226 18428 tablet_bootstrap.cc:492] T ea296376f7a94e938bd4e144f6becfb1 P 6cf2886496cc41f380499905bfd0cdbc: No bootstrap required, opened a new log
I20260927 12:44:47.352476 18428 ts_tablet_manager.cc:1397] T ea296376f7a94e938bd4e144f6becfb1 P 6cf2886496cc41f380499905bfd0cdbc: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 12:44:47.352844 18430 log.cc:824] T ea296376f7a94e938bd4e144f6becfb1 P 5c42b8c62b174737936e4dd6d9cba374: Log is configured to *not* fsync() on all Append() calls
I20260927 12:44:47.353489 18429 tablet_bootstrap.cc:492] T ea296376f7a94e938bd4e144f6becfb1 P d7e902c31074413b9dc80cb909e16de0: No bootstrap required, opened a new log
I20260927 12:44:47.353713 18429 ts_tablet_manager.cc:1397] T ea296376f7a94e938bd4e144f6becfb1 P d7e902c31074413b9dc80cb909e16de0: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 12:44:47.354423 18430 tablet_bootstrap.cc:492] T ea296376f7a94e938bd4e144f6becfb1 P 5c42b8c62b174737936e4dd6d9cba374: No bootstrap required, opened a new log
I20260927 12:44:47.354609 18430 ts_tablet_manager.cc:1397] T ea296376f7a94e938bd4e144f6becfb1 P 5c42b8c62b174737936e4dd6d9cba374: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 12:44:47.357081 18428 raft_consensus.cc:348] T ea296376f7a94e938bd4e144f6becfb1 P 6cf2886496cc41f380499905bfd0cdbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cf2886496cc41f380499905bfd0cdbc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45125 } } peers { permanent_uuid: "5c42b8c62b174737936e4dd6d9cba374" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42161 } } peers { permanent_uuid: "d7e902c31074413b9dc80cb909e16de0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40297 } }
I20260927 12:44:47.357370 18428 raft_consensus.cc:374] T ea296376f7a94e938bd4e144f6becfb1 P 6cf2886496cc41f380499905bfd0cdbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:44:47.357466 18428 raft_consensus.cc:729] T ea296376f7a94e938bd4e144f6becfb1 P 6cf2886496cc41f380499905bfd0cdbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cf2886496cc41f380499905bfd0cdbc, State: Initialized, Role: FOLLOWER
I20260927 12:44:47.357604 18430 raft_consensus.cc:348] T ea296376f7a94e938bd4e144f6becfb1 P 5c42b8c62b174737936e4dd6d9cba374 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cf2886496cc41f380499905bfd0cdbc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45125 } } peers { permanent_uuid: "5c42b8c62b174737936e4dd6d9cba374" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42161 } } peers { permanent_uuid: "d7e902c31074413b9dc80cb909e16de0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40297 } }
I20260927 12:44:47.357704 18429 raft_consensus.cc:348] T ea296376f7a94e938bd4e144f6becfb1 P d7e902c31074413b9dc80cb909e16de0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cf2886496cc41f380499905bfd0cdbc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45125 } } peers { permanent_uuid: "5c42b8c62b174737936e4dd6d9cba374" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42161 } } peers { permanent_uuid: "d7e902c31074413b9dc80cb909e16de0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40297 } }
I20260927 12:44:47.357885 18430 raft_consensus.cc:374] T ea296376f7a94e938bd4e144f6becfb1 P 5c42b8c62b174737936e4dd6d9cba374 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:44:47.357939 18430 raft_consensus.cc:729] T ea296376f7a94e938bd4e144f6becfb1 P 5c42b8c62b174737936e4dd6d9cba374 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c42b8c62b174737936e4dd6d9cba374, State: Initialized, Role: FOLLOWER
I20260927 12:44:47.357995 18429 raft_consensus.cc:374] T ea296376f7a94e938bd4e144f6becfb1 P d7e902c31074413b9dc80cb909e16de0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:44:47.358055 18429 raft_consensus.cc:729] T ea296376f7a94e938bd4e144f6becfb1 P d7e902c31074413b9dc80cb909e16de0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7e902c31074413b9dc80cb909e16de0, State: Initialized, Role: FOLLOWER
I20260927 12:44:47.357973 18428 consensus_queue.cc:260] T ea296376f7a94e938bd4e144f6becfb1 P 6cf2886496cc41f380499905bfd0cdbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cf2886496cc41f380499905bfd0cdbc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45125 } } peers { permanent_uuid: "5c42b8c62b174737936e4dd6d9cba374" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42161 } } peers { permanent_uuid: "d7e902c31074413b9dc80cb909e16de0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40297 } }
I20260927 12:44:47.358462 18430 consensus_queue.cc:260] T ea296376f7a94e938bd4e144f6becfb1 P 5c42b8c62b174737936e4dd6d9cba374 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cf2886496cc41f380499905bfd0cdbc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45125 } } peers { permanent_uuid: "5c42b8c62b174737936e4dd6d9cba374" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42161 } } peers { permanent_uuid: "d7e902c31074413b9dc80cb909e16de0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40297 } }
I20260927 12:44:47.358493 18429 consensus_queue.cc:260] T ea296376f7a94e938bd4e144f6becfb1 P d7e902c31074413b9dc80cb909e16de0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cf2886496cc41f380499905bfd0cdbc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45125 } } peers { permanent_uuid: "5c42b8c62b174737936e4dd6d9cba374" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42161 } } peers { permanent_uuid: "d7e902c31074413b9dc80cb909e16de0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40297 } }
I20260927 12:44:47.359041 18428 ts_tablet_manager.cc:1428] T ea296376f7a94e938bd4e144f6becfb1 P 6cf2886496cc41f380499905bfd0cdbc: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 12:44:47.359444 18430 ts_tablet_manager.cc:1428] T ea296376f7a94e938bd4e144f6becfb1 P 5c42b8c62b174737936e4dd6d9cba374: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 12:44:47.359449 18149 heartbeater.cc:503] Master 127.0.98.62:43091 was elected leader, sending a full tablet report...
I20260927 12:44:47.359513 18429 ts_tablet_manager.cc:1428] T ea296376f7a94e938bd4e144f6becfb1 P d7e902c31074413b9dc80cb909e16de0: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 12:44:47.359844 18280 heartbeater.cc:503] Master 127.0.98.62:43091 was elected leader, sending a full tablet report...
I20260927 12:44:47.359987 18411 heartbeater.cc:503] Master 127.0.98.62:43091 was elected leader, sending a full tablet report...
W20260927 12:44:47.408222 18150 tablet.cc:2369] T ea296376f7a94e938bd4e144f6becfb1 P 6cf2886496cc41f380499905bfd0cdbc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 12:44:47.538939 18412 tablet.cc:2369] T ea296376f7a94e938bd4e144f6becfb1 P 5c42b8c62b174737936e4dd6d9cba374: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:44:47.589462 18435 raft_consensus.cc:482] T ea296376f7a94e938bd4e144f6becfb1 P 5c42b8c62b174737936e4dd6d9cba374 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:44:47.589754 18435 raft_consensus.cc:504] T ea296376f7a94e938bd4e144f6becfb1 P 5c42b8c62b174737936e4dd6d9cba374 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cf2886496cc41f380499905bfd0cdbc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45125 } } peers { permanent_uuid: "5c42b8c62b174737936e4dd6d9cba374" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42161 } } peers { permanent_uuid: "d7e902c31074413b9dc80cb909e16de0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40297 } }
I20260927 12:44:47.590896 18435 leader_election.cc:288] T ea296376f7a94e938bd4e144f6becfb1 P 5c42b8c62b174737936e4dd6d9cba374 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6cf2886496cc41f380499905bfd0cdbc (127.0.98.1:45125), d7e902c31074413b9dc80cb909e16de0 (127.0.98.2:40297)
I20260927 12:44:47.594727 18413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:47.591250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45125 (local address 127.0.98.3:55025)
0927 12:44:47.591418 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:47.591439 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 12:44:47.591559 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:44:47.591896 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:44:47.591899 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:44:47.591919 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:44:47.592049 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:47.592056 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:47.592798 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:47.592802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:47.593678 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:47.593687 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:47.594301 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:47.594305 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:47.594402 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:47.594424 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 12:44:47.594518 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 12:44:47.594593 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260927 12:44:47.594733 18423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:47.591359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55025 (local address 127.0.98.1:45125)
0927 12:44:47.591488 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:47.591491 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:47.591506 (+ 15us) server_negotiation.cc:407] Connection header received
0927 12:44:47.591699 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:47.591702 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:47.591767 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:47.591863 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:44:47.592174 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:47.592677 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:47.593799 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:47.594179 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:47.594301 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:47.594362 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:47.594546 (+ 184us) server_negotiation.cc:299] Negotiation successful
0927 12:44:47.594612 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 12:44:47.595446 18103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea296376f7a94e938bd4e144f6becfb1" candidate_uuid: "5c42b8c62b174737936e4dd6d9cba374" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cf2886496cc41f380499905bfd0cdbc" is_pre_election: true
I20260927 12:44:47.595800 18437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:47.591597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40297 (local address 127.0.98.3:50565)
0927 12:44:47.592105 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:47.592134 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0927 12:44:47.592267 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:44:47.592583 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:44:47.592586 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:44:47.592614 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:44:47.592768 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:47.592774 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:47.593623 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:47.593628 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:47.594488 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:47.594497 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:47.595425 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:47.595428 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:47.595494 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:47.595510 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 12:44:47.595572 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 12:44:47.595616 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":159,"threads_started":1}
I20260927 12:44:47.595842 18427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:47.591755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50565 (local address 127.0.98.2:40297)
0927 12:44:47.591927 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:47.591936 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:47.592253 (+ 317us) server_negotiation.cc:407] Connection header received
0927 12:44:47.592410 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:47.592416 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:47.592471 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:47.592573 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:44:47.592904 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:47.593455 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:47.594643 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:47.595284 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:47.595414 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:47.595483 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:47.595611 (+ 128us) server_negotiation.cc:299] Negotiation successful
0927 12:44:47.595704 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 12:44:47.595861 18103 raft_consensus.cc:2454] T ea296376f7a94e938bd4e144f6becfb1 P 6cf2886496cc41f380499905bfd0cdbc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c42b8c62b174737936e4dd6d9cba374 in term 0.
I20260927 12:44:47.596447 18301 leader_election.cc:302] T ea296376f7a94e938bd4e144f6becfb1 P 5c42b8c62b174737936e4dd6d9cba374 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c42b8c62b174737936e4dd6d9cba374, 6cf2886496cc41f380499905bfd0cdbc; no voters:
I20260927 12:44:47.596463 18234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea296376f7a94e938bd4e144f6becfb1" candidate_uuid: "5c42b8c62b174737936e4dd6d9cba374" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7e902c31074413b9dc80cb909e16de0" is_pre_election: true
I20260927 12:44:47.596760 18234 raft_consensus.cc:2454] T ea296376f7a94e938bd4e144f6becfb1 P d7e902c31074413b9dc80cb909e16de0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c42b8c62b174737936e4dd6d9cba374 in term 0.
I20260927 12:44:47.596788 18435 raft_consensus.cc:2781] T ea296376f7a94e938bd4e144f6becfb1 P 5c42b8c62b174737936e4dd6d9cba374 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:44:47.596913 18435 raft_consensus.cc:482] T ea296376f7a94e938bd4e144f6becfb1 P 5c42b8c62b174737936e4dd6d9cba374 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:44:47.596987 18435 raft_consensus.cc:3037] T ea296376f7a94e938bd4e144f6becfb1 P 5c42b8c62b174737936e4dd6d9cba374 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:44:47.598248 18435 raft_consensus.cc:504] T ea296376f7a94e938bd4e144f6becfb1 P 5c42b8c62b174737936e4dd6d9cba374 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cf2886496cc41f380499905bfd0cdbc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45125 } } peers { permanent_uuid: "5c42b8c62b174737936e4dd6d9cba374" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42161 } } peers { permanent_uuid: "d7e902c31074413b9dc80cb909e16de0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40297 } }
I20260927 12:44:47.598688 18435 leader_election.cc:288] T ea296376f7a94e938bd4e144f6becfb1 P 5c42b8c62b174737936e4dd6d9cba374 [CANDIDATE]: Term 1 election: Requested vote from peers 6cf2886496cc41f380499905bfd0cdbc (127.0.98.1:45125), d7e902c31074413b9dc80cb909e16de0 (127.0.98.2:40297)
I20260927 12:44:47.599067 18103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea296376f7a94e938bd4e144f6becfb1" candidate_uuid: "5c42b8c62b174737936e4dd6d9cba374" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cf2886496cc41f380499905bfd0cdbc"
I20260927 12:44:47.599138 18234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea296376f7a94e938bd4e144f6becfb1" candidate_uuid: "5c42b8c62b174737936e4dd6d9cba374" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7e902c31074413b9dc80cb909e16de0"
I20260927 12:44:47.599220 18103 raft_consensus.cc:3037] T ea296376f7a94e938bd4e144f6becfb1 P 6cf2886496cc41f380499905bfd0cdbc [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:44:47.599264 18234 raft_consensus.cc:3037] T ea296376f7a94e938bd4e144f6becfb1 P d7e902c31074413b9dc80cb909e16de0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:44:47.600639 18234 raft_consensus.cc:2454] T ea296376f7a94e938bd4e144f6becfb1 P d7e902c31074413b9dc80cb909e16de0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c42b8c62b174737936e4dd6d9cba374 in term 1.
I20260927 12:44:47.600647 18103 raft_consensus.cc:2454] T ea296376f7a94e938bd4e144f6becfb1 P 6cf2886496cc41f380499905bfd0cdbc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c42b8c62b174737936e4dd6d9cba374 in term 1.
I20260927 12:44:47.600994 18301 leader_election.cc:302] T ea296376f7a94e938bd4e144f6becfb1 P 5c42b8c62b174737936e4dd6d9cba374 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c42b8c62b174737936e4dd6d9cba374, 6cf2886496cc41f380499905bfd0cdbc; no voters:
W20260927 12:44:47.601025 18281 tablet.cc:2369] T ea296376f7a94e938bd4e144f6becfb1 P d7e902c31074413b9dc80cb909e16de0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:44:47.601209 18435 raft_consensus.cc:2781] T ea296376f7a94e938bd4e144f6becfb1 P 5c42b8c62b174737936e4dd6d9cba374 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:44:47.601482 18435 raft_consensus.cc:686] T ea296376f7a94e938bd4e144f6becfb1 P 5c42b8c62b174737936e4dd6d9cba374 [term 1 LEADER]: Becoming Leader. State: Replica: 5c42b8c62b174737936e4dd6d9cba374, State: Running, Role: LEADER
I20260927 12:44:47.601919 18435 consensus_queue.cc:237] T ea296376f7a94e938bd4e144f6becfb1 P 5c42b8c62b174737936e4dd6d9cba374 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cf2886496cc41f380499905bfd0cdbc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45125 } } peers { permanent_uuid: "5c42b8c62b174737936e4dd6d9cba374" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42161 } } peers { permanent_uuid: "d7e902c31074413b9dc80cb909e16de0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40297 } }
I20260927 12:44:47.605428 17959 catalog_manager.cc:5495] T ea296376f7a94e938bd4e144f6becfb1 P 5c42b8c62b174737936e4dd6d9cba374 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5c42b8c62b174737936e4dd6d9cba374 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "5c42b8c62b174737936e4dd6d9cba374" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cf2886496cc41f380499905bfd0cdbc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c42b8c62b174737936e4dd6d9cba374" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42161 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7e902c31074413b9dc80cb909e16de0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40297 } health_report { overall_health: UNKNOWN } } }
I20260927 12:44:47.629000 18425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:47.625363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38648 (local address 127.0.98.3:42161)
0927 12:44:47.625494 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:47.625497 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:47.625511 (+ 14us) server_negotiation.cc:407] Connection header received
0927 12:44:47.625668 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:47.625674 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:47.625749 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:47.625881 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:44:47.626089 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:47.626628 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:47.627522 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:47.627758 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:47.627839 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:47.628176 (+ 337us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:44:47.628594 (+ 418us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:44:47.628683 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:47.628795 (+ 112us) server_negotiation.cc:299] Negotiation successful
0927 12:44:47.628838 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 12:44:47.635249 18103 raft_consensus.cc:1261] T ea296376f7a94e938bd4e144f6becfb1 P 6cf2886496cc41f380499905bfd0cdbc [term 1 FOLLOWER]: Refusing update from remote peer 5c42b8c62b174737936e4dd6d9cba374: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:44:47.635249 18234 raft_consensus.cc:1261] T ea296376f7a94e938bd4e144f6becfb1 P d7e902c31074413b9dc80cb909e16de0 [term 1 FOLLOWER]: Refusing update from remote peer 5c42b8c62b174737936e4dd6d9cba374: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:44:47.636233 18438 consensus_queue.cc:1035] T ea296376f7a94e938bd4e144f6becfb1 P 5c42b8c62b174737936e4dd6d9cba374 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7e902c31074413b9dc80cb909e16de0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:44:47.636447 18435 consensus_queue.cc:1035] T ea296376f7a94e938bd4e144f6becfb1 P 5c42b8c62b174737936e4dd6d9cba374 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cf2886496cc41f380499905bfd0cdbc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:44:47.644325 18442 mvcc.cc:205] Tried to move back new op lower bound from 7333941606945570816 to 7333941606820106240. Current Snapshot: MvccSnapshot[applied={T|T < 7333941606945570816}]
I20260927 12:44:47.646641 18443 mvcc.cc:205] Tried to move back new op lower bound from 7333941606945570816 to 7333941606820106240. Current Snapshot: MvccSnapshot[applied={T|T < 7333941606945570816}]
I20260927 12:44:47.651098 17959 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39820:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:44:47.651252 17959 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39820:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:44:47.653452 17959 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6852b84b02b84ea5a3a77204476ae7a5: Sending DeleteTablet for 3 replicas of tablet ea296376f7a94e938bd4e144f6becfb1
I20260927 12:44:47.654166 18083 tablet_service.cc:1514] Processing DeleteTablet for tablet ea296376f7a94e938bd4e144f6becfb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:44:47 UTC) from {username='slave'} at 127.0.0.1:41216
I20260927 12:44:47.654326 18345 tablet_service.cc:1514] Processing DeleteTablet for tablet ea296376f7a94e938bd4e144f6becfb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:44:47 UTC) from {username='slave'} at 127.0.0.1:38634
I20260927 12:44:47.654953 18451 tablet_replica.cc:330] T ea296376f7a94e938bd4e144f6becfb1 P 5c42b8c62b174737936e4dd6d9cba374: stopping tablet replica
I20260927 12:44:47.654909 18214 tablet_service.cc:1514] Processing DeleteTablet for tablet ea296376f7a94e938bd4e144f6becfb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:44:47 UTC) from {username='slave'} at 127.0.0.1:57080
I20260927 12:44:47.655002 18450 tablet_replica.cc:330] T ea296376f7a94e938bd4e144f6becfb1 P 6cf2886496cc41f380499905bfd0cdbc: stopping tablet replica
I20260927 12:44:47.655288 18451 raft_consensus.cc:2229] T ea296376f7a94e938bd4e144f6becfb1 P 5c42b8c62b174737936e4dd6d9cba374 [term 1 LEADER]: Raft consensus shutting down.
I20260927 12:44:47.655364 18450 raft_consensus.cc:2229] T ea296376f7a94e938bd4e144f6becfb1 P 6cf2886496cc41f380499905bfd0cdbc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:44:47.655646 18451 raft_consensus.cc:2258] T ea296376f7a94e938bd4e144f6becfb1 P 5c42b8c62b174737936e4dd6d9cba374 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:44:47.655791 18450 raft_consensus.cc:2258] T ea296376f7a94e938bd4e144f6becfb1 P 6cf2886496cc41f380499905bfd0cdbc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:44:47.656049 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 18021
I20260927 12:44:47.657917 18452 tablet_replica.cc:330] T ea296376f7a94e938bd4e144f6becfb1 P d7e902c31074413b9dc80cb909e16de0: stopping tablet replica
I20260927 12:44:47.658262 18452 raft_consensus.cc:2229] T ea296376f7a94e938bd4e144f6becfb1 P d7e902c31074413b9dc80cb909e16de0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:44:47.658617 18452 raft_consensus.cc:2258] T ea296376f7a94e938bd4e144f6becfb1 P d7e902c31074413b9dc80cb909e16de0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:44:47.659051 18451 ts_tablet_manager.cc:1905] T ea296376f7a94e938bd4e144f6becfb1 P 5c42b8c62b174737936e4dd6d9cba374: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:44:47.661383 18451 ts_tablet_manager.cc:1918] T ea296376f7a94e938bd4e144f6becfb1 P 5c42b8c62b174737936e4dd6d9cba374: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:44:47.661530 18451 log.cc:1196] T ea296376f7a94e938bd4e144f6becfb1 P 5c42b8c62b174737936e4dd6d9cba374: Deleting WAL directory at /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790512840836255-392-0/minicluster-data/ts-2/wal/wals/ea296376f7a94e938bd4e144f6becfb1
I20260927 12:44:47.661957 18451 ts_tablet_manager.cc:1939] T ea296376f7a94e938bd4e144f6becfb1 P 5c42b8c62b174737936e4dd6d9cba374: Deleting consensus metadata
I20260927 12:44:47.663323 17948 catalog_manager.cc:4841] TS 5c42b8c62b174737936e4dd6d9cba374 (127.0.98.3:42161): tablet ea296376f7a94e938bd4e144f6becfb1 (table test-table [id=f9eaa9ce655b4dd3bdc743bd6f61542f]) successfully deleted
W20260927 12:44:47.663604 17948 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45125 (error 108)
I20260927 12:44:47.663686 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 18152
I20260927 12:44:47.665776 18424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:47.665359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45125 (local address 127.0.0.1:41228)
0927 12:44:47.665617 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:47.665688 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151}
W20260927 12:44:47.666069 17948 catalog_manager.cc:4584] TS 6cf2886496cc41f380499905bfd0cdbc (127.0.98.1:45125): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ea296376f7a94e938bd4e144f6becfb1: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45125: connect: Connection refused (error 111)
I20260927 12:44:47.670991 18424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:47.670728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40297 (local address 127.0.0.1:57090)
0927 12:44:47.670854 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:47.670911 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:40297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 12:44:47.671070 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 18283
I20260927 12:44:47.677152 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 17930
2026-09-27T12:44:47Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1238 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:44:47 dist-test-slave-bnc1 krb5kdc[18456](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:44:47 dist-test-slave-bnc1 krb5kdc[18456](info): set up 2 sockets
Sep 27 12:44:47 dist-test-slave-bnc1 krb5kdc[18456](info): commencing operation
krb5kdc: starting...
W20260927 12:44:48.713912 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/krb5kdc/test-user.keytab.
Sep 27 12:44:48 dist-test-slave-bnc1 krb5kdc[18456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513088, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:44:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:44:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:44:48.840886 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36229
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36229
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:44:48.950060 18469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:44:48.950378 18469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:44:48.950470 18469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:44:48.954514 18469 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:44:48.954603 18469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:44:48.954643 18469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:44:48.954681 18469 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:44:48.954710 18469 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:44:48.959976 18469 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36229
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:44:48.961097 18469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:44:48.962222 18469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:44:48.970985 18475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:44:48.971099 18469 server_base.cc:1034] running on GCE node
W20260927 12:44:48.970966 18474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:44:48.970985 18477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:44:48.971849 18469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:44:48.972939 18469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:44:48.974119 18469 hybrid_clock.cc:648] HybridClock initialized: now 1790513088974102 us; error 29 us; skew 500 ppm
Sep 27 12:44:48 dist-test-slave-bnc1 krb5kdc[18456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513088, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:44:48.976877 18469 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:44:48.977810 18469 webserver.cc:460] Webserver started at http://127.0.98.62:39271/ using document root <none> and password file <none>
I20260927 12:44:48.978332 18469 fs_manager.cc:359] Metadata directory not provided
I20260927 12:44:48.978389 18469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:44:48.978549 18469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:44:48.980501 18469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/minicluster-data/master-0/data/instance:
uuid: "6f02aee494324d989383aac1c31e27b3"
format_stamp: "Formatted at 2026-09-27 12:44:48 on dist-test-slave-bnc1"
server_key: "68934ffcf53e8edd97aebe2ae8ac8d59"
server_key_iv: "2a419fe91771bf81e585a9da99da4920"
server_key_version: "encryptionkey@0"
I20260927 12:44:48.981021 18469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/minicluster-data/master-0/wal/instance:
uuid: "6f02aee494324d989383aac1c31e27b3"
format_stamp: "Formatted at 2026-09-27 12:44:48 on dist-test-slave-bnc1"
server_key: "68934ffcf53e8edd97aebe2ae8ac8d59"
server_key_iv: "2a419fe91771bf81e585a9da99da4920"
server_key_version: "encryptionkey@0"
I20260927 12:44:48.984741 18469 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 12:44:48.987175 18484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:44:48.988391 18469 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 12:44:48.988488 18469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/minicluster-data/master-0/wal
uuid: "6f02aee494324d989383aac1c31e27b3"
format_stamp: "Formatted at 2026-09-27 12:44:48 on dist-test-slave-bnc1"
server_key: "68934ffcf53e8edd97aebe2ae8ac8d59"
server_key_iv: "2a419fe91771bf81e585a9da99da4920"
server_key_version: "encryptionkey@0"
I20260927 12:44:48.988576 18469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:44:49.007664 18469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:44:49.011116 18469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:44:49.011276 18469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:44:49.018633 18469 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36229
I20260927 12:44:49.018652 18536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36229 every 8 connection(s)
I20260927 12:44:49.019626 18469 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:44:49.022425 18537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:44:49.027212 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 18469
I20260927 12:44:49.027371 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/minicluster-data/master-0/wal/instance
I20260927 12:44:49.027670 392 external_mini_cluster.cc:1428] Setting key 42b965d6df14a4f7bd849400c286a773
I20260927 12:44:49.028321 18537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f02aee494324d989383aac1c31e27b3: Bootstrap starting.
I20260927 12:44:49.030853 18537 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f02aee494324d989383aac1c31e27b3: Neither blocks nor log segments found. Creating new log.
I20260927 12:44:49.031875 18537 log.cc:824] T 00000000000000000000000000000000 P 6f02aee494324d989383aac1c31e27b3: Log is configured to *not* fsync() on all Append() calls
I20260927 12:44:49.034152 18537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f02aee494324d989383aac1c31e27b3: No bootstrap required, opened a new log
Sep 27 12:44:49 dist-test-slave-bnc1 krb5kdc[18456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513088, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:44:49.038527 18537 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f02aee494324d989383aac1c31e27b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f02aee494324d989383aac1c31e27b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36229 } }
I20260927 12:44:49.038782 18537 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f02aee494324d989383aac1c31e27b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:44:49.038906 18537 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f02aee494324d989383aac1c31e27b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f02aee494324d989383aac1c31e27b3, State: Initialized, Role: FOLLOWER
I20260927 12:44:49.039448 18537 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f02aee494324d989383aac1c31e27b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f02aee494324d989383aac1c31e27b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36229 } }
I20260927 12:44:49.039616 18537 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f02aee494324d989383aac1c31e27b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:44:49.039687 18537 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f02aee494324d989383aac1c31e27b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:44:49.039806 18537 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f02aee494324d989383aac1c31e27b3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:44:49.040948 18537 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f02aee494324d989383aac1c31e27b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f02aee494324d989383aac1c31e27b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36229 } }
I20260927 12:44:49.041316 18537 leader_election.cc:302] T 00000000000000000000000000000000 P 6f02aee494324d989383aac1c31e27b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f02aee494324d989383aac1c31e27b3; no voters:
I20260927 12:44:49.041712 18537 leader_election.cc:288] T 00000000000000000000000000000000 P 6f02aee494324d989383aac1c31e27b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:44:49.041776 18542 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f02aee494324d989383aac1c31e27b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:44:49.042055 18542 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f02aee494324d989383aac1c31e27b3 [term 1 LEADER]: Becoming Leader. State: Replica: 6f02aee494324d989383aac1c31e27b3, State: Running, Role: LEADER
I20260927 12:44:49.042430 18542 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f02aee494324d989383aac1c31e27b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f02aee494324d989383aac1c31e27b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36229 } }
I20260927 12:44:49.043107 18540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:49.029154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41370 (local address 127.0.98.62:36229)
0927 12:44:49.029639 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:49.029655 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:49.029691 (+ 36us) server_negotiation.cc:407] Connection header received
0927 12:44:49.030357 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:49.030375 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:49.030776 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:49.031121 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:44:49.032099 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:49.033161 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:49.034387 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:49.034870 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:49.034999 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:49.037820 (+ 2821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:44:49.037825 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:44:49.037839 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:44:49.037865 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:44:49.039872 (+ 2007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:44:49.040362 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:44:49.040368 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:44:49.040374 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:44:49.040513 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:44:49.040779 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:44:49.040785 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:44:49.040790 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:44:49.041344 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:44:49.041535 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:49.041928 (+ 393us) server_negotiation.cc:299] Negotiation successful
0927 12:44:49.042260 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":129,"threads_started":1}
I20260927 12:44:49.043131 18537 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f02aee494324d989383aac1c31e27b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:44:49.043643 18542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f02aee494324d989383aac1c31e27b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f02aee494324d989383aac1c31e27b3. Latest consensus state: current_term: 1 leader_uuid: "6f02aee494324d989383aac1c31e27b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f02aee494324d989383aac1c31e27b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36229 } } }
I20260927 12:44:49.043758 18542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f02aee494324d989383aac1c31e27b3 [sys.catalog]: This master's current role is: LEADER
I20260927 12:44:49.043962 18542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f02aee494324d989383aac1c31e27b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f02aee494324d989383aac1c31e27b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f02aee494324d989383aac1c31e27b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36229 } } }
I20260927 12:44:49.044067 18542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f02aee494324d989383aac1c31e27b3 [sys.catalog]: This master's current role is: LEADER
W20260927 12:44:49.048193 18557 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6f02aee494324d989383aac1c31e27b3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 12:44:49.048286 18557 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 12:44:49.048355 18547 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:44:49.049049 18547 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:44:49.055078 18547 catalog_manager.cc:1347] Generated new cluster ID: 16fa9efdd6d34f26b56c2ea1d122a210
I20260927 12:44:49.055155 18547 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:44:49.076061 18547 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:44:49.077075 18547 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:44:49.091688 18547 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f02aee494324d989383aac1c31e27b3: Generated new TSK 0
I20260927 12:44:49.092517 18547 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 12:44:49.124994 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36229
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:44:49.239902 18563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:44:49.240226 18563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:44:49.240309 18563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:44:49.244632 18563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:44:49.244720 18563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:44:49.244831 18563 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:44:49.250366 18563 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18563
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:44:49.251446 18563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:44:49.252473 18563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:44:49.262151 18571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:44:49.262197 18569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:44:49.262193 18568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:44:49.262243 18563 server_base.cc:1034] running on GCE node
I20260927 12:44:49.262859 18563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:44:49.263499 18563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:44:49.264678 18563 hybrid_clock.cc:648] HybridClock initialized: now 1790513089264648 us; error 46 us; skew 500 ppm
Sep 27 12:44:49 dist-test-slave-bnc1 krb5kdc[18456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513089, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:44:49.267474 18563 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 12:44:49.268378 18563 webserver.cc:460] Webserver started at http://127.0.98.1:36325/ using document root <none> and password file <none>
I20260927 12:44:49.268946 18563 fs_manager.cc:359] Metadata directory not provided
I20260927 12:44:49.269035 18563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:44:49.269265 18563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:44:49.271401 18563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/minicluster-data/ts-0/data/instance:
uuid: "296021576c3549a2b905097f0b46d853"
format_stamp: "Formatted at 2026-09-27 12:44:49 on dist-test-slave-bnc1"
server_key: "0b3af63dec16949fdc607ccc1257add0"
server_key_iv: "46c8a598a5103f7dd71abb8613d50391"
server_key_version: "encryptionkey@0"
I20260927 12:44:49.271960 18563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/minicluster-data/ts-0/wal/instance:
uuid: "296021576c3549a2b905097f0b46d853"
format_stamp: "Formatted at 2026-09-27 12:44:49 on dist-test-slave-bnc1"
server_key: "0b3af63dec16949fdc607ccc1257add0"
server_key_iv: "46c8a598a5103f7dd71abb8613d50391"
server_key_version: "encryptionkey@0"
I20260927 12:44:49.275557 18563 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 12:44:49.278117 18578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:44:49.279433 18563 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:44:49.279575 18563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/minicluster-data/ts-0/wal
uuid: "296021576c3549a2b905097f0b46d853"
format_stamp: "Formatted at 2026-09-27 12:44:49 on dist-test-slave-bnc1"
server_key: "0b3af63dec16949fdc607ccc1257add0"
server_key_iv: "46c8a598a5103f7dd71abb8613d50391"
server_key_version: "encryptionkey@0"
I20260927 12:44:49.279688 18563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:44:49.295981 18563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:44:49.299209 18563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:44:49.299418 18563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:44:49.300041 18563 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:44:49.300890 18563 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:44:49.300935 18563 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:44:49.301004 18563 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:44:49.301034 18563 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:44:49.310202 18563 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45595
I20260927 12:44:49.310218 18691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45595 every 8 connection(s)
I20260927 12:44:49.311174 18563 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:44:49.311702 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 18563
I20260927 12:44:49.311854 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/minicluster-data/ts-0/wal/instance
I20260927 12:44:49.312175 392 external_mini_cluster.cc:1428] Setting key 2110dc17c63cbeb5f64a56e6387d87fa
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Sep 27 12:44:49 dist-test-slave-bnc1 krb5kdc[18456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513089, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:44:49.331327 18540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:49.313145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36575 (local address 127.0.98.62:36229)
0927 12:44:49.313276 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:49.313287 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:49.314617 (+ 1330us) server_negotiation.cc:407] Connection header received
0927 12:44:49.316418 (+ 1801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:49.316424 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:49.316495 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:49.316603 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:44:49.318339 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:49.319004 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:49.320990 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:49.321270 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:49.321359 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:49.327019 (+ 5660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:44:49.327027 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:44:49.327033 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:44:49.327076 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:44:49.328735 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:44:49.329461 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:44:49.329464 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:44:49.329465 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:44:49.329555 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:44:49.330022 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:44:49.330025 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:44:49.330026 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:44:49.330262 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:44:49.330367 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:49.331079 (+ 712us) server_negotiation.cc:299] Negotiation successful
0927 12:44:49.331198 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 12:44:49.332661 18694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:49.313793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36229 (local address 127.0.98.1:36575)
0927 12:44:49.314425 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:49.314489 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 12:44:49.316165 (+ 1676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:44:49.316772 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:44:49.316779 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:44:49.317359 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:44:49.318126 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:49.318150 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:49.319153 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:49.319160 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:49.320824 (+ 1664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:49.320853 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:49.321456 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:49.321460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:49.321725 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:49.322486 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:44:49.322509 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:44:49.326794 (+ 4285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:44:49.328866 (+ 2072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:44:49.328877 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:44:49.328898 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:44:49.329335 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:44:49.329702 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:44:49.329708 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:44:49.329713 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:44:49.329910 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:44:49.330391 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:44:49.330400 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:44:49.330761 (+ 361us) client_negotiation.cc:769] Sending connection context
0927 12:44:49.330988 (+ 227us) client_negotiation.cc:240] Negotiation successful
0927 12:44:49.331426 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":141,"threads_started":1}
I20260927 12:44:49.334046 18692 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36229
I20260927 12:44:49.334347 18692 heartbeater.cc:464] Registering TS with master...
I20260927 12:44:49.334860 18692 heartbeater.cc:511] Master 127.0.98.62:36229 requested a full tablet report, sending...
I20260927 12:44:49.336489 18500 ts_manager.cc:194] Registered new tserver with Master: 296021576c3549a2b905097f0b46d853 (127.0.98.1:45595)
I20260927 12:44:49.337893 18500 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36575
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 12:44:49.345804 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36229
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:44:49.457528 18697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:44:49.457878 18697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:44:49.457973 18697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:44:49.462018 18697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:44:49.462111 18697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:44:49.462205 18697 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:44:49.467401 18697 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:44:49.468457 18697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:44:49.469542 18697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:44:49.479672 18702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:44:49.479697 18703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:44:49.479734 18697 server_base.cc:1034] running on GCE node
W20260927 12:44:49.479691 18705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:44:49.480347 18697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:44:49.480986 18697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:44:49.482162 18697 hybrid_clock.cc:648] HybridClock initialized: now 1790513089482140 us; error 36 us; skew 500 ppm
Sep 27 12:44:49 dist-test-slave-bnc1 krb5kdc[18456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513089, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:44:49.485085 18697 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 12:44:49.486017 18697 webserver.cc:460] Webserver started at http://127.0.98.2:39263/ using document root <none> and password file <none>
I20260927 12:44:49.486573 18697 fs_manager.cc:359] Metadata directory not provided
I20260927 12:44:49.486656 18697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:44:49.486871 18697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:44:49.488911 18697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/minicluster-data/ts-1/data/instance:
uuid: "056e698fd3864a7cb3d3f8f700d04b77"
format_stamp: "Formatted at 2026-09-27 12:44:49 on dist-test-slave-bnc1"
server_key: "bd2d97df5bcf6be867bbf17ef1015da7"
server_key_iv: "0cc630205f2863ae1176a08fd264df53"
server_key_version: "encryptionkey@0"
I20260927 12:44:49.489502 18697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/minicluster-data/ts-1/wal/instance:
uuid: "056e698fd3864a7cb3d3f8f700d04b77"
format_stamp: "Formatted at 2026-09-27 12:44:49 on dist-test-slave-bnc1"
server_key: "bd2d97df5bcf6be867bbf17ef1015da7"
server_key_iv: "0cc630205f2863ae1176a08fd264df53"
server_key_version: "encryptionkey@0"
I20260927 12:44:49.493336 18697 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 12:44:49.495888 18712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:44:49.497291 18697 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 12:44:49.497438 18697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/minicluster-data/ts-1/wal
uuid: "056e698fd3864a7cb3d3f8f700d04b77"
format_stamp: "Formatted at 2026-09-27 12:44:49 on dist-test-slave-bnc1"
server_key: "bd2d97df5bcf6be867bbf17ef1015da7"
server_key_iv: "0cc630205f2863ae1176a08fd264df53"
server_key_version: "encryptionkey@0"
I20260927 12:44:49.497558 18697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:44:49.514720 18697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:44:49.517905 18697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:44:49.518105 18697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:44:49.518707 18697 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:44:49.519541 18697 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:44:49.519611 18697 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:44:49.519682 18697 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:44:49.519738 18697 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:44:49.528710 18697 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42481
I20260927 12:44:49.528723 18825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42481 every 8 connection(s)
I20260927 12:44:49.529633 18697 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:44:49.532135 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 18697
I20260927 12:44:49.532275 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/minicluster-data/ts-1/wal/instance
I20260927 12:44:49.532582 392 external_mini_cluster.cc:1428] Setting key 9707bdf571e541c24d91db54db2b778d
Sep 27 12:44:49 dist-test-slave-bnc1 krb5kdc[18456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513089, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260927 12:44:49.543839 18540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:49.531380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42237 (local address 127.0.98.62:36229)
0927 12:44:49.531548 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:49.531553 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:49.532539 (+ 986us) server_negotiation.cc:407] Connection header received
0927 12:44:49.534161 (+ 1622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:49.534164 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:49.534227 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:49.534335 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:44:49.535483 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:49.536004 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:49.537144 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:49.537371 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:49.537481 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:49.539921 (+ 2440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:44:49.539924 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:44:49.539926 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:44:49.539949 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:44:49.541413 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:44:49.542043 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:44:49.542046 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:44:49.542047 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:44:49.542125 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:44:49.542516 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:44:49.542519 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:44:49.542521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:44:49.542736 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:44:49.542844 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:49.543503 (+ 659us) server_negotiation.cc:299] Negotiation successful
0927 12:44:49.543643 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 12:44:49.544941 18828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:49.531809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36229 (local address 127.0.98.2:42237)
0927 12:44:49.532390 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:49.532432 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 12:44:49.533943 (+ 1511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:44:49.534514 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:44:49.534521 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:44:49.534875 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:44:49.535341 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:49.535355 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:49.536126 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:49.536130 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:49.537017 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:49.537026 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:49.537516 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:49.537520 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:49.537733 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:49.538327 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:44:49.538343 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:44:49.539752 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:44:49.541551 (+ 1799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:44:49.541557 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:44:49.541594 (+ 37us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:44:49.541918 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:44:49.542244 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:44:49.542246 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:44:49.542247 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:44:49.542402 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:44:49.542841 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:44:49.542848 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:44:49.543090 (+ 242us) client_negotiation.cc:769] Sending connection context
0927 12:44:49.543414 (+ 324us) client_negotiation.cc:240] Negotiation successful
0927 12:44:49.543753 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":136,"threads_started":1}
I20260927 12:44:49.546254 18826 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36229
I20260927 12:44:49.546500 18826 heartbeater.cc:464] Registering TS with master...
I20260927 12:44:49.547003 18826 heartbeater.cc:511] Master 127.0.98.62:36229 requested a full tablet report, sending...
I20260927 12:44:49.548094 18500 ts_manager.cc:194] Registered new tserver with Master: 056e698fd3864a7cb3d3f8f700d04b77 (127.0.98.2:42481)
I20260927 12:44:49.548697 18500 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42237
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 12:44:49.560855 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36229
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:44:49.671870 18831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:44:49.672132 18831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:44:49.672175 18831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:44:49.676247 18831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:44:49.676303 18831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:44:49.676362 18831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:44:49.681440 18831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:44:49.682504 18831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:44:49.683570 18831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:44:49.693234 18837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:44:49.693511 18831 server_base.cc:1034] running on GCE node
W20260927 12:44:49.693181 18839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:44:49.693181 18836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:44:49.694072 18831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:44:49.694669 18831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:44:49.695829 18831 hybrid_clock.cc:648] HybridClock initialized: now 1790513089695813 us; error 27 us; skew 500 ppm
Sep 27 12:44:49 dist-test-slave-bnc1 krb5kdc[18456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513089, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:44:49.698648 18831 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 12:44:49.699549 18831 webserver.cc:460] Webserver started at http://127.0.98.3:46583/ using document root <none> and password file <none>
I20260927 12:44:49.700063 18831 fs_manager.cc:359] Metadata directory not provided
I20260927 12:44:49.700120 18831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:44:49.700292 18831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:44:49.702510 18831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/minicluster-data/ts-2/data/instance:
uuid: "8c2c7cc458344b6db77f1df2537e7e14"
format_stamp: "Formatted at 2026-09-27 12:44:49 on dist-test-slave-bnc1"
server_key: "549b8f8fdf90308f79f92a1688647d78"
server_key_iv: "9e6f5c1b67ac82139adf9885d7f96d3d"
server_key_version: "encryptionkey@0"
I20260927 12:44:49.703019 18831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8c2c7cc458344b6db77f1df2537e7e14"
format_stamp: "Formatted at 2026-09-27 12:44:49 on dist-test-slave-bnc1"
server_key: "549b8f8fdf90308f79f92a1688647d78"
server_key_iv: "9e6f5c1b67ac82139adf9885d7f96d3d"
server_key_version: "encryptionkey@0"
I20260927 12:44:49.707652 18831 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260927 12:44:49.710232 18846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:44:49.711614 18831 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:44:49.711752 18831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/minicluster-data/ts-2/wal
uuid: "8c2c7cc458344b6db77f1df2537e7e14"
format_stamp: "Formatted at 2026-09-27 12:44:49 on dist-test-slave-bnc1"
server_key: "549b8f8fdf90308f79f92a1688647d78"
server_key_iv: "9e6f5c1b67ac82139adf9885d7f96d3d"
server_key_version: "encryptionkey@0"
I20260927 12:44:49.711869 18831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:44:49.724265 18831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:44:49.727339 18831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:44:49.727567 18831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:44:49.728188 18831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:44:49.729043 18831 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:44:49.729122 18831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:44:49.729218 18831 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:44:49.729269 18831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:44:49.738435 18831 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35685
I20260927 12:44:49.738452 18959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35685 every 8 connection(s)
I20260927 12:44:49.739362 18831 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:44:49.747236 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 18831
I20260927 12:44:49.747394 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/minicluster-data/ts-2/wal/instance
I20260927 12:44:49.747705 392 external_mini_cluster.cc:1428] Setting key 7eb1a5a5f5ba1aa553d3003ca24e5752
Sep 27 12:44:49 dist-test-slave-bnc1 krb5kdc[18456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513089, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:44:49.753959 18540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:49.741238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34153 (local address 127.0.98.62:36229)
0927 12:44:49.741378 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:49.741381 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:49.742278 (+ 897us) server_negotiation.cc:407] Connection header received
0927 12:44:49.743678 (+ 1400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:49.743682 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:49.743741 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:49.743825 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:44:49.745148 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:49.745780 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:49.746924 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:49.747281 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:49.747405 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:49.750396 (+ 2991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:44:49.750400 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:44:49.750402 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:44:49.750433 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:44:49.751883 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:44:49.752408 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:44:49.752412 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:44:49.752414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:44:49.752494 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:44:49.752838 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:44:49.752843 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:44:49.752845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:44:49.753027 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:44:49.753130 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:49.753646 (+ 516us) server_negotiation.cc:299] Negotiation successful
0927 12:44:49.753768 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 12:44:49.754781 18962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:49.741682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36229 (local address 127.0.98.3:34153)
0927 12:44:49.742124 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:49.742168 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 12:44:49.743467 (+ 1299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:44:49.744051 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:44:49.744062 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:44:49.744506 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:44:49.745001 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:49.745014 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:49.745904 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:49.745910 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:49.746794 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:49.746803 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:49.747494 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:49.747500 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:49.747683 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:49.748351 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:44:49.748368 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:44:49.750228 (+ 1860us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:44:49.752014 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:44:49.752019 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:44:49.752035 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:44:49.752300 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:44:49.752582 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:44:49.752587 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:44:49.752589 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:44:49.752742 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:44:49.753129 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:44:49.753137 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:44:49.753389 (+ 252us) client_negotiation.cc:769] Sending connection context
0927 12:44:49.753600 (+ 211us) client_negotiation.cc:240] Negotiation successful
0927 12:44:49.753957 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":111,"threads_started":1}
I20260927 12:44:49.755913 18960 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36229
I20260927 12:44:49.756141 18960 heartbeater.cc:464] Registering TS with master...
I20260927 12:44:49.756610 18960 heartbeater.cc:511] Master 127.0.98.62:36229 requested a full tablet report, sending...
I20260927 12:44:49.757704 18500 ts_manager.cc:194] Registered new tserver with Master: 8c2c7cc458344b6db77f1df2537e7e14 (127.0.98.3:35685)
I20260927 12:44:49.758275 18500 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34153
I20260927 12:44:49.761695 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 12:44:49.771687 18540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:49.763609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41378 (local address 127.0.98.62:36229)
0927 12:44:49.763741 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:49.763747 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:49.763992 (+ 245us) server_negotiation.cc:407] Connection header received
0927 12:44:49.764125 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:49.764128 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:49.764173 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:49.764261 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:44:49.764947 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:49.765472 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:49.766345 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:49.766559 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:49.766667 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:49.768277 (+ 1610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:44:49.768280 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:44:49.768283 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:44:49.768312 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:44:49.769885 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:44:49.770382 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:44:49.770385 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:44:49.770387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:44:49.770469 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:44:49.770793 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:44:49.770796 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:44:49.770798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:44:49.771006 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:44:49.771162 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:49.771432 (+ 270us) server_negotiation.cc:299] Negotiation successful
0927 12:44:49.771523 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 12:44:49.774585 18500 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41378:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 12:44:49.777684 18500 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:44:49.791960 18971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:49.786004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45595 (local address 127.0.0.1:51376)
0927 12:44:49.786445 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:49.786492 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 12:44:49.787159 (+ 667us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:44:49.787698 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:44:49.787704 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:44:49.787726 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:44:49.787873 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:49.787883 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:49.789042 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:49.789048 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:49.790425 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:49.790438 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:49.791506 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:49.791509 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:49.791598 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:49.791620 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 12:44:49.791665 (+ 45us) client_negotiation.cc:240] Negotiation successful
0927 12:44:49.791779 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":87,"threads_started":1}
I20260927 12:44:49.792136 18976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:49.787033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48404 (local address 127.0.98.2:42481)
0927 12:44:49.787452 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:49.787461 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:49.787479 (+ 18us) server_negotiation.cc:407] Connection header received
0927 12:44:49.787539 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:49.787543 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:49.787678 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:49.787841 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:44:49.788073 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:49.789223 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:49.790635 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:49.791226 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:49.791336 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:49.791478 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:49.791873 (+ 395us) server_negotiation.cc:299] Negotiation successful
0927 12:44:49.791968 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":123,"threads_started":1}
I20260927 12:44:49.792344 18972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:49.786111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51376 (local address 127.0.98.1:45595)
0927 12:44:49.786717 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:49.786736 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:49.786763 (+ 27us) server_negotiation.cc:407] Connection header received
0927 12:44:49.787325 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:49.787333 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:49.787531 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:49.787692 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:44:49.788022 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:49.788930 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:49.790559 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:49.791383 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:49.791901 (+ 518us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:49.792041 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:49.792094 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 12:44:49.792190 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":71,"threads_started":1}
I20260927 12:44:49.792856 18973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:49.786676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35685 (local address 127.0.0.1:60064)
0927 12:44:49.787957 (+ 1281us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:49.787972 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:44:49.788059 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:44:49.788552 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:44:49.788555 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:44:49.788574 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:44:49.788656 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:49.788661 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:49.789979 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:49.789986 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:49.791106 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:49.791117 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:49.792424 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:49.792428 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:49.792529 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:49.792552 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 12:44:49.792653 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 12:44:49.792717 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1209,"mutex_wait_us":3,"thread_start_us":92,"threads_started":1}
I20260927 12:44:49.793066 18975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:49.786855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60064 (local address 127.0.98.3:35685)
0927 12:44:49.787400 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:49.787411 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:49.788152 (+ 741us) server_negotiation.cc:407] Connection header received
0927 12:44:49.788238 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:49.788243 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:49.788408 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:49.788554 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:44:49.788778 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:49.789833 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:49.791275 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:49.792125 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:49.792234 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:49.792357 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:49.792807 (+ 450us) server_negotiation.cc:299] Negotiation successful
0927 12:44:49.792904 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":152,"threads_started":1}
I20260927 12:44:49.793543 18974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:49.786100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42481 (local address 127.0.0.1:48404)
0927 12:44:49.786989 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:49.787011 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 12:44:49.787099 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:44:49.787817 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:44:49.787820 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:44:49.787838 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:44:49.787944 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:49.787952 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:49.789352 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:49.789356 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:49.790519 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:49.790528 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:49.791333 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:49.791337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:49.791444 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:49.791479 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 12:44:49.791603 (+ 124us) client_negotiation.cc:240] Negotiation successful
0927 12:44:49.791750 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":817,"mutex_wait_us":24,"thread_start_us":111,"threads_started":1}
I20260927 12:44:49.795318 18626 tablet_service.cc:1467] Processing CreateTablet for tablet bffe71c638f940118b7a30b9855544d3 (DEFAULT_TABLE table=test-table [id=852fd249ffdb4dd89fe31270b0bcaa19]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:44:49.795790 18894 tablet_service.cc:1467] Processing CreateTablet for tablet bffe71c638f940118b7a30b9855544d3 (DEFAULT_TABLE table=test-table [id=852fd249ffdb4dd89fe31270b0bcaa19]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:44:49.796301 18626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bffe71c638f940118b7a30b9855544d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:44:49.796087 18760 tablet_service.cc:1467] Processing CreateTablet for tablet bffe71c638f940118b7a30b9855544d3 (DEFAULT_TABLE table=test-table [id=852fd249ffdb4dd89fe31270b0bcaa19]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:44:49.796844 18894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bffe71c638f940118b7a30b9855544d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:44:49.796931 18760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bffe71c638f940118b7a30b9855544d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:44:49.802901 18977 tablet_bootstrap.cc:492] T bffe71c638f940118b7a30b9855544d3 P 296021576c3549a2b905097f0b46d853: Bootstrap starting.
I20260927 12:44:49.804515 18978 tablet_bootstrap.cc:492] T bffe71c638f940118b7a30b9855544d3 P 056e698fd3864a7cb3d3f8f700d04b77: Bootstrap starting.
I20260927 12:44:49.805023 18977 tablet_bootstrap.cc:654] T bffe71c638f940118b7a30b9855544d3 P 296021576c3549a2b905097f0b46d853: Neither blocks nor log segments found. Creating new log.
I20260927 12:44:49.805887 18979 tablet_bootstrap.cc:492] T bffe71c638f940118b7a30b9855544d3 P 8c2c7cc458344b6db77f1df2537e7e14: Bootstrap starting.
I20260927 12:44:49.806160 18977 log.cc:824] T bffe71c638f940118b7a30b9855544d3 P 296021576c3549a2b905097f0b46d853: Log is configured to *not* fsync() on all Append() calls
I20260927 12:44:49.806407 18978 tablet_bootstrap.cc:654] T bffe71c638f940118b7a30b9855544d3 P 056e698fd3864a7cb3d3f8f700d04b77: Neither blocks nor log segments found. Creating new log.
I20260927 12:44:49.807619 18978 log.cc:824] T bffe71c638f940118b7a30b9855544d3 P 056e698fd3864a7cb3d3f8f700d04b77: Log is configured to *not* fsync() on all Append() calls
I20260927 12:44:49.808028 18979 tablet_bootstrap.cc:654] T bffe71c638f940118b7a30b9855544d3 P 8c2c7cc458344b6db77f1df2537e7e14: Neither blocks nor log segments found. Creating new log.
I20260927 12:44:49.808218 18977 tablet_bootstrap.cc:492] T bffe71c638f940118b7a30b9855544d3 P 296021576c3549a2b905097f0b46d853: No bootstrap required, opened a new log
I20260927 12:44:49.808421 18977 ts_tablet_manager.cc:1397] T bffe71c638f940118b7a30b9855544d3 P 296021576c3549a2b905097f0b46d853: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260927 12:44:49.809175 18979 log.cc:824] T bffe71c638f940118b7a30b9855544d3 P 8c2c7cc458344b6db77f1df2537e7e14: Log is configured to *not* fsync() on all Append() calls
I20260927 12:44:49.809703 18978 tablet_bootstrap.cc:492] T bffe71c638f940118b7a30b9855544d3 P 056e698fd3864a7cb3d3f8f700d04b77: No bootstrap required, opened a new log
I20260927 12:44:49.809916 18978 ts_tablet_manager.cc:1397] T bffe71c638f940118b7a30b9855544d3 P 056e698fd3864a7cb3d3f8f700d04b77: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 12:44:49.811070 18979 tablet_bootstrap.cc:492] T bffe71c638f940118b7a30b9855544d3 P 8c2c7cc458344b6db77f1df2537e7e14: No bootstrap required, opened a new log
I20260927 12:44:49.811283 18979 ts_tablet_manager.cc:1397] T bffe71c638f940118b7a30b9855544d3 P 8c2c7cc458344b6db77f1df2537e7e14: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260927 12:44:49.811630 18977 raft_consensus.cc:348] T bffe71c638f940118b7a30b9855544d3 P 296021576c3549a2b905097f0b46d853 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296021576c3549a2b905097f0b46d853" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45595 } } peers { permanent_uuid: "056e698fd3864a7cb3d3f8f700d04b77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42481 } } peers { permanent_uuid: "8c2c7cc458344b6db77f1df2537e7e14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35685 } }
I20260927 12:44:49.811868 18977 raft_consensus.cc:374] T bffe71c638f940118b7a30b9855544d3 P 296021576c3549a2b905097f0b46d853 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:44:49.811940 18977 raft_consensus.cc:729] T bffe71c638f940118b7a30b9855544d3 P 296021576c3549a2b905097f0b46d853 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 296021576c3549a2b905097f0b46d853, State: Initialized, Role: FOLLOWER
I20260927 12:44:49.812456 18977 consensus_queue.cc:260] T bffe71c638f940118b7a30b9855544d3 P 296021576c3549a2b905097f0b46d853 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296021576c3549a2b905097f0b46d853" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45595 } } peers { permanent_uuid: "056e698fd3864a7cb3d3f8f700d04b77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42481 } } peers { permanent_uuid: "8c2c7cc458344b6db77f1df2537e7e14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35685 } }
I20260927 12:44:49.813090 18692 heartbeater.cc:503] Master 127.0.98.62:36229 was elected leader, sending a full tablet report...
I20260927 12:44:49.813710 18978 raft_consensus.cc:348] T bffe71c638f940118b7a30b9855544d3 P 056e698fd3864a7cb3d3f8f700d04b77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296021576c3549a2b905097f0b46d853" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45595 } } peers { permanent_uuid: "056e698fd3864a7cb3d3f8f700d04b77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42481 } } peers { permanent_uuid: "8c2c7cc458344b6db77f1df2537e7e14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35685 } }
I20260927 12:44:49.814026 18978 raft_consensus.cc:374] T bffe71c638f940118b7a30b9855544d3 P 056e698fd3864a7cb3d3f8f700d04b77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:44:49.814114 18978 raft_consensus.cc:729] T bffe71c638f940118b7a30b9855544d3 P 056e698fd3864a7cb3d3f8f700d04b77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 056e698fd3864a7cb3d3f8f700d04b77, State: Initialized, Role: FOLLOWER
I20260927 12:44:49.814560 18978 consensus_queue.cc:260] T bffe71c638f940118b7a30b9855544d3 P 056e698fd3864a7cb3d3f8f700d04b77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296021576c3549a2b905097f0b46d853" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45595 } } peers { permanent_uuid: "056e698fd3864a7cb3d3f8f700d04b77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42481 } } peers { permanent_uuid: "8c2c7cc458344b6db77f1df2537e7e14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35685 } }
I20260927 12:44:49.814881 18977 ts_tablet_manager.cc:1428] T bffe71c638f940118b7a30b9855544d3 P 296021576c3549a2b905097f0b46d853: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260927 12:44:49.815037 18979 raft_consensus.cc:348] T bffe71c638f940118b7a30b9855544d3 P 8c2c7cc458344b6db77f1df2537e7e14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296021576c3549a2b905097f0b46d853" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45595 } } peers { permanent_uuid: "056e698fd3864a7cb3d3f8f700d04b77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42481 } } peers { permanent_uuid: "8c2c7cc458344b6db77f1df2537e7e14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35685 } }
I20260927 12:44:49.815299 18979 raft_consensus.cc:374] T bffe71c638f940118b7a30b9855544d3 P 8c2c7cc458344b6db77f1df2537e7e14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:44:49.815387 18979 raft_consensus.cc:729] T bffe71c638f940118b7a30b9855544d3 P 8c2c7cc458344b6db77f1df2537e7e14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c2c7cc458344b6db77f1df2537e7e14, State: Initialized, Role: FOLLOWER
I20260927 12:44:49.815610 18978 ts_tablet_manager.cc:1428] T bffe71c638f940118b7a30b9855544d3 P 056e698fd3864a7cb3d3f8f700d04b77: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260927 12:44:49.815778 18979 consensus_queue.cc:260] T bffe71c638f940118b7a30b9855544d3 P 8c2c7cc458344b6db77f1df2537e7e14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296021576c3549a2b905097f0b46d853" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45595 } } peers { permanent_uuid: "056e698fd3864a7cb3d3f8f700d04b77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42481 } } peers { permanent_uuid: "8c2c7cc458344b6db77f1df2537e7e14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35685 } }
I20260927 12:44:49.815968 18826 heartbeater.cc:503] Master 127.0.98.62:36229 was elected leader, sending a full tablet report...
I20260927 12:44:49.816499 18960 heartbeater.cc:503] Master 127.0.98.62:36229 was elected leader, sending a full tablet report...
I20260927 12:44:49.816746 18979 ts_tablet_manager.cc:1428] T bffe71c638f940118b7a30b9855544d3 P 8c2c7cc458344b6db77f1df2537e7e14: Time spent starting tablet: real 0.005s user 0.000s sys 0.006s
I20260927 12:44:49.835736 18984 raft_consensus.cc:482] T bffe71c638f940118b7a30b9855544d3 P 056e698fd3864a7cb3d3f8f700d04b77 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:44:49.835924 18984 raft_consensus.cc:504] T bffe71c638f940118b7a30b9855544d3 P 056e698fd3864a7cb3d3f8f700d04b77 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296021576c3549a2b905097f0b46d853" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45595 } } peers { permanent_uuid: "056e698fd3864a7cb3d3f8f700d04b77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42481 } } peers { permanent_uuid: "8c2c7cc458344b6db77f1df2537e7e14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35685 } }
I20260927 12:44:49.836997 18984 leader_election.cc:288] T bffe71c638f940118b7a30b9855544d3 P 056e698fd3864a7cb3d3f8f700d04b77 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 296021576c3549a2b905097f0b46d853 (127.0.98.1:45595), 8c2c7cc458344b6db77f1df2537e7e14 (127.0.98.3:35685)
I20260927 12:44:49.841012 18972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:49.837335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58033 (local address 127.0.98.1:45595)
0927 12:44:49.837451 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:49.837456 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:49.837529 (+ 73us) server_negotiation.cc:407] Connection header received
0927 12:44:49.837762 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:49.837765 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:49.837815 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:49.837888 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:44:49.838295 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:49.838856 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:49.840025 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:49.840413 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:49.840493 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:49.840579 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:49.840789 (+ 210us) server_negotiation.cc:299] Negotiation successful
0927 12:44:49.840881 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 12:44:49.841042 18828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:49.837251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45595 (local address 127.0.98.2:58033)
0927 12:44:49.837457 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:49.837473 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 12:44:49.837621 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:44:49.837950 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:44:49.837953 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:44:49.837970 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:44:49.838173 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:49.838181 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:49.838969 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:49.838972 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:49.839903 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:49.839912 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:49.840535 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:49.840538 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:49.840653 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:49.840689 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 12:44:49.840750 (+ 61us) client_negotiation.cc:240] Negotiation successful
0927 12:44:49.840903 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":93}
I20260927 12:44:49.841042 18975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:49.837335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34717 (local address 127.0.98.3:35685)
0927 12:44:49.837451 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:49.837456 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:49.837811 (+ 355us) server_negotiation.cc:407] Connection header received
0927 12:44:49.837908 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:49.837913 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:49.837970 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:49.838059 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:44:49.838456 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:49.838964 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:49.840031 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:49.840435 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:49.840553 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:49.840624 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:49.840849 (+ 225us) server_negotiation.cc:299] Negotiation successful
0927 12:44:49.840926 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 12:44:49.841068 18986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:49.837251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35685 (local address 127.0.98.2:34717)
0927 12:44:49.837662 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:49.837693 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 12:44:49.837799 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:44:49.838080 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:44:49.838082 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:44:49.838096 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:44:49.838338 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:49.838345 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:49.839070 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:49.839073 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:49.839906 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:49.839915 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:49.840575 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:49.840579 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:49.840687 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:49.840705 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 12:44:49.840802 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 12:44:49.840921 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"mutex_wait_us":178,"thread_start_us":106,"threads_started":1}
I20260927 12:44:49.841712 18914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bffe71c638f940118b7a30b9855544d3" candidate_uuid: "056e698fd3864a7cb3d3f8f700d04b77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c2c7cc458344b6db77f1df2537e7e14" is_pre_election: true
I20260927 12:44:49.841719 18646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bffe71c638f940118b7a30b9855544d3" candidate_uuid: "056e698fd3864a7cb3d3f8f700d04b77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "296021576c3549a2b905097f0b46d853" is_pre_election: true
I20260927 12:44:49.841996 18914 raft_consensus.cc:2454] T bffe71c638f940118b7a30b9855544d3 P 8c2c7cc458344b6db77f1df2537e7e14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 056e698fd3864a7cb3d3f8f700d04b77 in term 0.
I20260927 12:44:49.842012 18646 raft_consensus.cc:2454] T bffe71c638f940118b7a30b9855544d3 P 296021576c3549a2b905097f0b46d853 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 056e698fd3864a7cb3d3f8f700d04b77 in term 0.
I20260927 12:44:49.842468 18715 leader_election.cc:302] T bffe71c638f940118b7a30b9855544d3 P 056e698fd3864a7cb3d3f8f700d04b77 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 056e698fd3864a7cb3d3f8f700d04b77, 8c2c7cc458344b6db77f1df2537e7e14; no voters:
I20260927 12:44:49.842747 18984 raft_consensus.cc:2781] T bffe71c638f940118b7a30b9855544d3 P 056e698fd3864a7cb3d3f8f700d04b77 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:44:49.842829 18984 raft_consensus.cc:482] T bffe71c638f940118b7a30b9855544d3 P 056e698fd3864a7cb3d3f8f700d04b77 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:44:49.842871 18984 raft_consensus.cc:3037] T bffe71c638f940118b7a30b9855544d3 P 056e698fd3864a7cb3d3f8f700d04b77 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:44:49.844008 18984 raft_consensus.cc:504] T bffe71c638f940118b7a30b9855544d3 P 056e698fd3864a7cb3d3f8f700d04b77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296021576c3549a2b905097f0b46d853" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45595 } } peers { permanent_uuid: "056e698fd3864a7cb3d3f8f700d04b77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42481 } } peers { permanent_uuid: "8c2c7cc458344b6db77f1df2537e7e14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35685 } }
I20260927 12:44:49.844372 18984 leader_election.cc:288] T bffe71c638f940118b7a30b9855544d3 P 056e698fd3864a7cb3d3f8f700d04b77 [CANDIDATE]: Term 1 election: Requested vote from peers 296021576c3549a2b905097f0b46d853 (127.0.98.1:45595), 8c2c7cc458344b6db77f1df2537e7e14 (127.0.98.3:35685)
I20260927 12:44:49.844696 18914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bffe71c638f940118b7a30b9855544d3" candidate_uuid: "056e698fd3864a7cb3d3f8f700d04b77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c2c7cc458344b6db77f1df2537e7e14"
I20260927 12:44:49.844735 18646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bffe71c638f940118b7a30b9855544d3" candidate_uuid: "056e698fd3864a7cb3d3f8f700d04b77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "296021576c3549a2b905097f0b46d853"
I20260927 12:44:49.844798 18914 raft_consensus.cc:3037] T bffe71c638f940118b7a30b9855544d3 P 8c2c7cc458344b6db77f1df2537e7e14 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:44:49.844882 18646 raft_consensus.cc:3037] T bffe71c638f940118b7a30b9855544d3 P 296021576c3549a2b905097f0b46d853 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:44:49.845943 18914 raft_consensus.cc:2454] T bffe71c638f940118b7a30b9855544d3 P 8c2c7cc458344b6db77f1df2537e7e14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 056e698fd3864a7cb3d3f8f700d04b77 in term 1.
I20260927 12:44:49.846079 18646 raft_consensus.cc:2454] T bffe71c638f940118b7a30b9855544d3 P 296021576c3549a2b905097f0b46d853 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 056e698fd3864a7cb3d3f8f700d04b77 in term 1.
I20260927 12:44:49.846290 18715 leader_election.cc:302] T bffe71c638f940118b7a30b9855544d3 P 056e698fd3864a7cb3d3f8f700d04b77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 056e698fd3864a7cb3d3f8f700d04b77, 8c2c7cc458344b6db77f1df2537e7e14; no voters:
I20260927 12:44:49.846449 18984 raft_consensus.cc:2781] T bffe71c638f940118b7a30b9855544d3 P 056e698fd3864a7cb3d3f8f700d04b77 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:44:49.846692 18984 raft_consensus.cc:686] T bffe71c638f940118b7a30b9855544d3 P 056e698fd3864a7cb3d3f8f700d04b77 [term 1 LEADER]: Becoming Leader. State: Replica: 056e698fd3864a7cb3d3f8f700d04b77, State: Running, Role: LEADER
I20260927 12:44:49.847030 18984 consensus_queue.cc:237] T bffe71c638f940118b7a30b9855544d3 P 056e698fd3864a7cb3d3f8f700d04b77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296021576c3549a2b905097f0b46d853" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45595 } } peers { permanent_uuid: "056e698fd3864a7cb3d3f8f700d04b77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42481 } } peers { permanent_uuid: "8c2c7cc458344b6db77f1df2537e7e14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35685 } }
I20260927 12:44:49.850262 18499 catalog_manager.cc:5495] T bffe71c638f940118b7a30b9855544d3 P 056e698fd3864a7cb3d3f8f700d04b77 reported cstate change: term changed from 0 to 1, leader changed from <none> to 056e698fd3864a7cb3d3f8f700d04b77 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "056e698fd3864a7cb3d3f8f700d04b77" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "296021576c3549a2b905097f0b46d853" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "056e698fd3864a7cb3d3f8f700d04b77" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c2c7cc458344b6db77f1df2537e7e14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35685 } health_report { overall_health: UNKNOWN } } }
I20260927 12:44:49.872532 18976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:49.868960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48416 (local address 127.0.98.2:42481)
0927 12:44:49.869117 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:49.869123 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:49.869145 (+ 22us) server_negotiation.cc:407] Connection header received
0927 12:44:49.869212 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:49.869214 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:49.869267 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:49.869381 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:44:49.869659 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:49.870176 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:49.871033 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:49.871251 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:49.871405 (+ 154us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:49.871727 (+ 322us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:44:49.872122 (+ 395us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:44:49.872279 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:49.872340 (+ 61us) server_negotiation.cc:299] Negotiation successful
0927 12:44:49.872374 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 12:44:49.878566 18646 raft_consensus.cc:1261] T bffe71c638f940118b7a30b9855544d3 P 296021576c3549a2b905097f0b46d853 [term 1 FOLLOWER]: Refusing update from remote peer 056e698fd3864a7cb3d3f8f700d04b77: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:44:49.878572 18914 raft_consensus.cc:1261] T bffe71c638f940118b7a30b9855544d3 P 8c2c7cc458344b6db77f1df2537e7e14 [term 1 FOLLOWER]: Refusing update from remote peer 056e698fd3864a7cb3d3f8f700d04b77: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:44:49.879323 18987 consensus_queue.cc:1035] T bffe71c638f940118b7a30b9855544d3 P 056e698fd3864a7cb3d3f8f700d04b77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c2c7cc458344b6db77f1df2537e7e14" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:44:49.879590 18984 consensus_queue.cc:1035] T bffe71c638f940118b7a30b9855544d3 P 056e698fd3864a7cb3d3f8f700d04b77 [LEADER]: Connected to new peer: Peer: permanent_uuid: "296021576c3549a2b905097f0b46d853" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:44:49.886591 18990 mvcc.cc:205] Tried to move back new op lower bound from 7333941616134619136 to 7333941616016584704. Current Snapshot: MvccSnapshot[applied={T|T < 7333941616134619136}]
I20260927 12:44:49.886862 18992 mvcc.cc:205] Tried to move back new op lower bound from 7333941616134619136 to 7333941616016584704. Current Snapshot: MvccSnapshot[applied={T|T < 7333941616134619136}]
I20260927 12:44:49.887753 18991 mvcc.cc:205] Tried to move back new op lower bound from 7333941616134619136 to 7333941616016584704. Current Snapshot: MvccSnapshot[applied={T|T < 7333941616134619136}]
I20260927 12:44:49.896766 18500 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41378:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:44:49.896977 18500 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41378:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:44:49.899578 18500 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6f02aee494324d989383aac1c31e27b3: Sending DeleteTablet for 3 replicas of tablet bffe71c638f940118b7a30b9855544d3
I20260927 12:44:49.900451 18626 tablet_service.cc:1514] Processing DeleteTablet for tablet bffe71c638f940118b7a30b9855544d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:44:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51376
I20260927 12:44:49.900614 18894 tablet_service.cc:1514] Processing DeleteTablet for tablet bffe71c638f940118b7a30b9855544d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:44:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60064
I20260927 12:44:49.900777 18760 tablet_service.cc:1514] Processing DeleteTablet for tablet bffe71c638f940118b7a30b9855544d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:44:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48404
I20260927 12:44:49.901386 19002 tablet_replica.cc:330] T bffe71c638f940118b7a30b9855544d3 P 056e698fd3864a7cb3d3f8f700d04b77: stopping tablet replica
I20260927 12:44:49.901938 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 18563
I20260927 12:44:49.901829 19002 raft_consensus.cc:2229] T bffe71c638f940118b7a30b9855544d3 P 056e698fd3864a7cb3d3f8f700d04b77 [term 1 LEADER]: Raft consensus shutting down.
I20260927 12:44:49.904120 19002 raft_consensus.cc:2258] T bffe71c638f940118b7a30b9855544d3 P 056e698fd3864a7cb3d3f8f700d04b77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:44:49.905454 19002 ts_tablet_manager.cc:1905] T bffe71c638f940118b7a30b9855544d3 P 056e698fd3864a7cb3d3f8f700d04b77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:44:49.908084 19002 ts_tablet_manager.cc:1918] T bffe71c638f940118b7a30b9855544d3 P 056e698fd3864a7cb3d3f8f700d04b77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:44:49.908241 19002 log.cc:1196] T bffe71c638f940118b7a30b9855544d3 P 056e698fd3864a7cb3d3f8f700d04b77: Deleting WAL directory at /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/minicluster-data/ts-1/wal/wals/bffe71c638f940118b7a30b9855544d3
I20260927 12:44:49.908919 19002 ts_tablet_manager.cc:1939] T bffe71c638f940118b7a30b9855544d3 P 056e698fd3864a7cb3d3f8f700d04b77: Deleting consensus metadata
I20260927 12:44:49.910439 18487 catalog_manager.cc:4841] TS 056e698fd3864a7cb3d3f8f700d04b77 (127.0.98.2:42481): tablet bffe71c638f940118b7a30b9855544d3 (table test-table [id=852fd249ffdb4dd89fe31270b0bcaa19]) successfully deleted
I20260927 12:44:49.910892 19001 tablet_replica.cc:330] T bffe71c638f940118b7a30b9855544d3 P 8c2c7cc458344b6db77f1df2537e7e14: stopping tablet replica
I20260927 12:44:49.911289 19001 raft_consensus.cc:2229] T bffe71c638f940118b7a30b9855544d3 P 8c2c7cc458344b6db77f1df2537e7e14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:44:49.911646 19001 raft_consensus.cc:2258] T bffe71c638f940118b7a30b9855544d3 P 8c2c7cc458344b6db77f1df2537e7e14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:44:49.912995 19001 ts_tablet_manager.cc:1905] T bffe71c638f940118b7a30b9855544d3 P 8c2c7cc458344b6db77f1df2537e7e14: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260927 12:44:49.913551 18485 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45595 (error 108)
I20260927 12:44:49.916352 19001 ts_tablet_manager.cc:1918] T bffe71c638f940118b7a30b9855544d3 P 8c2c7cc458344b6db77f1df2537e7e14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:44:49.916450 19001 log.cc:1196] T bffe71c638f940118b7a30b9855544d3 P 8c2c7cc458344b6db77f1df2537e7e14: Deleting WAL directory at /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790512840836255-392-0/minicluster-data/ts-2/wal/wals/bffe71c638f940118b7a30b9855544d3
I20260927 12:44:49.916800 19001 ts_tablet_manager.cc:1939] T bffe71c638f940118b7a30b9855544d3 P 8c2c7cc458344b6db77f1df2537e7e14: Deleting consensus metadata
I20260927 12:44:49.916859 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 18697
I20260927 12:44:49.918591 18487 catalog_manager.cc:4841] TS 8c2c7cc458344b6db77f1df2537e7e14 (127.0.98.3:35685): tablet bffe71c638f940118b7a30b9855544d3 (table test-table [id=852fd249ffdb4dd89fe31270b0bcaa19]) successfully deleted
I20260927 12:44:49.919044 18974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:49.918751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45595 (local address 127.0.0.1:51380)
0927 12:44:49.918864 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:49.918943 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260927 12:44:49.919376 18485 catalog_manager.cc:4584] TS 296021576c3549a2b905097f0b46d853 (127.0.98.1:45595): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bffe71c638f940118b7a30b9855544d3: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45595: connect: Connection refused (error 111)
I20260927 12:44:49.924544 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 18831
I20260927 12:44:49.930150 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 18469
2026-09-27T12:44:49Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2252 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-27T12:44:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:44:49Z Disabled control of system clock
I20260927 12:44:49.960915 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512840836255-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512840836255-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512840836255-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35335
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35335
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 12:44:50.075035 19007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:44:50.075307 19007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:44:50.075344 19007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:44:50.079406 19007 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:44:50.079468 19007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:44:50.079483 19007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:44:50.079500 19007 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:44:50.079516 19007 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:44:50.084183 19007 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512840836255-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512840836255-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35335
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512840836255-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:44:50.085279 19007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:44:50.086715 19007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:44:50.095358 19013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:44:50.095347 19015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:44:50.095489 19007 server_base.cc:1034] running on GCE node
W20260927 12:44:50.095677 19012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:44:50.096213 19007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:44:50.097298 19007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:44:50.098457 19007 hybrid_clock.cc:648] HybridClock initialized: now 1790513090098437 us; error 32 us; skew 500 ppm
I20260927 12:44:50.100219 19007 webserver.cc:460] Webserver started at http://127.0.98.62:33345/ using document root <none> and password file <none>
I20260927 12:44:50.100754 19007 fs_manager.cc:359] Metadata directory not provided
I20260927 12:44:50.100814 19007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:44:50.100999 19007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:44:50.103016 19007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512840836255-392-0/minicluster-data/master-0/data/instance:
uuid: "3863be4dc4774eb3a2f220a905630c92"
format_stamp: "Formatted at 2026-09-27 12:44:50 on dist-test-slave-bnc1"
server_key: "ecfa79e8ca85f0ac545ac0655cb692c1"
server_key_iv: "92573e0f5215f9f1472eea2212b88d8b"
server_key_version: "encryptionkey@0"
I20260927 12:44:50.103533 19007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512840836255-392-0/minicluster-data/master-0/wal/instance:
uuid: "3863be4dc4774eb3a2f220a905630c92"
format_stamp: "Formatted at 2026-09-27 12:44:50 on dist-test-slave-bnc1"
server_key: "ecfa79e8ca85f0ac545ac0655cb692c1"
server_key_iv: "92573e0f5215f9f1472eea2212b88d8b"
server_key_version: "encryptionkey@0"
I20260927 12:44:50.107247 19007 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 12:44:50.109697 19021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:44:50.110894 19007 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 12:44:50.110996 19007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512840836255-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512840836255-392-0/minicluster-data/master-0/wal
uuid: "3863be4dc4774eb3a2f220a905630c92"
format_stamp: "Formatted at 2026-09-27 12:44:50 on dist-test-slave-bnc1"
server_key: "ecfa79e8ca85f0ac545ac0655cb692c1"
server_key_iv: "92573e0f5215f9f1472eea2212b88d8b"
server_key_version: "encryptionkey@0"
I20260927 12:44:50.111079 19007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512840836255-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512840836255-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512840836255-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:44:50.128165 19007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:44:50.128882 19007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:44:50.129038 19007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:44:50.136008 19007 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35335
I20260927 12:44:50.136015 19073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35335 every 8 connection(s)
I20260927 12:44:50.136989 19007 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:44:50.140165 19074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:44:50.145973 19074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3863be4dc4774eb3a2f220a905630c92: Bootstrap starting.
I20260927 12:44:50.147110 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 19007
I20260927 12:44:50.147251 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512840836255-392-0/minicluster-data/master-0/wal/instance
I20260927 12:44:50.147536 392 external_mini_cluster.cc:1428] Setting key c6d053c2e0afda867e70ea4f769cb8eb
I20260927 12:44:50.148798 19074 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3863be4dc4774eb3a2f220a905630c92: Neither blocks nor log segments found. Creating new log.
I20260927 12:44:50.149526 19074 log.cc:824] T 00000000000000000000000000000000 P 3863be4dc4774eb3a2f220a905630c92: Log is configured to *not* fsync() on all Append() calls
I20260927 12:44:50.151646 19074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3863be4dc4774eb3a2f220a905630c92: No bootstrap required, opened a new log
I20260927 12:44:50.155984 19074 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3863be4dc4774eb3a2f220a905630c92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3863be4dc4774eb3a2f220a905630c92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35335 } }
I20260927 12:44:50.156227 19074 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3863be4dc4774eb3a2f220a905630c92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:44:50.156307 19074 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3863be4dc4774eb3a2f220a905630c92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3863be4dc4774eb3a2f220a905630c92, State: Initialized, Role: FOLLOWER
I20260927 12:44:50.156783 19074 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3863be4dc4774eb3a2f220a905630c92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3863be4dc4774eb3a2f220a905630c92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35335 } }
I20260927 12:44:50.156931 19074 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3863be4dc4774eb3a2f220a905630c92 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:44:50.157011 19077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:50.148730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41656 (local address 127.0.98.62:35335)
0927 12:44:50.149425 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:50.149440 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:50.149476 (+ 36us) server_negotiation.cc:407] Connection header received
0927 12:44:50.150200 (+ 724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:50.150219 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:50.150549 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:50.150827 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:44:50.151157 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:50.152204 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:50.153462 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:50.154003 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:50.154123 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:50.155050 (+ 927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:44:50.155056 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:44:50.155069 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:44:50.155484 (+ 415us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:44:50.155541 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:44:50.155560 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:44:50.155636 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:50.155802 (+ 166us) server_negotiation.cc:299] Negotiation successful
0927 12:44:50.156106 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":161,"threads_started":1}
I20260927 12:44:50.157018 19074 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3863be4dc4774eb3a2f220a905630c92 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:44:50.157212 19074 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3863be4dc4774eb3a2f220a905630c92 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:44:50.158353 19074 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3863be4dc4774eb3a2f220a905630c92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3863be4dc4774eb3a2f220a905630c92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35335 } }
I20260927 12:44:50.158695 19074 leader_election.cc:302] T 00000000000000000000000000000000 P 3863be4dc4774eb3a2f220a905630c92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3863be4dc4774eb3a2f220a905630c92; no voters:
I20260927 12:44:50.159025 19074 leader_election.cc:288] T 00000000000000000000000000000000 P 3863be4dc4774eb3a2f220a905630c92 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:44:50.159116 19079 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3863be4dc4774eb3a2f220a905630c92 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:44:50.159354 19079 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3863be4dc4774eb3a2f220a905630c92 [term 1 LEADER]: Becoming Leader. State: Replica: 3863be4dc4774eb3a2f220a905630c92, State: Running, Role: LEADER
I20260927 12:44:50.159662 19079 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3863be4dc4774eb3a2f220a905630c92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3863be4dc4774eb3a2f220a905630c92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35335 } }
I20260927 12:44:50.160274 19074 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3863be4dc4774eb3a2f220a905630c92 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:44:50.161553 19080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3863be4dc4774eb3a2f220a905630c92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3863be4dc4774eb3a2f220a905630c92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3863be4dc4774eb3a2f220a905630c92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35335 } } }
I20260927 12:44:50.161720 19080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3863be4dc4774eb3a2f220a905630c92 [sys.catalog]: This master's current role is: LEADER
I20260927 12:44:50.161620 19081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3863be4dc4774eb3a2f220a905630c92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3863be4dc4774eb3a2f220a905630c92. Latest consensus state: current_term: 1 leader_uuid: "3863be4dc4774eb3a2f220a905630c92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3863be4dc4774eb3a2f220a905630c92" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35335 } } }
I20260927 12:44:50.161926 19081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3863be4dc4774eb3a2f220a905630c92 [sys.catalog]: This master's current role is: LEADER
I20260927 12:44:50.162248 19087 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:44:50.164914 19087 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:44:50.170405 19087 catalog_manager.cc:1347] Generated new cluster ID: 50da3d82e9d24f95b7a6ddb6feacd891
I20260927 12:44:50.170486 19087 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:44:50.189844 19087 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:44:50.191135 19087 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:44:50.196403 19087 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3863be4dc4774eb3a2f220a905630c92: Generated new TSK 0
I20260927 12:44:50.197218 19087 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:44:50.207957 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512840836255-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512840836255-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512840836255-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35335
--builtin_ntp_servers=127.0.98.20:39865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 12:44:50.319291 19098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:44:50.319591 19098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:44:50.319681 19098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:44:50.324137 19098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:44:50.324244 19098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:44:50.324343 19098 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:44:50.329288 19098 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512840836255-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512840836255-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19098
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512840836255-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:44:50.330478 19098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:44:50.331858 19098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:44:50.341502 19106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:44:50.341543 19098 server_base.cc:1034] running on GCE node
W20260927 12:44:50.341485 19103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:44:50.341478 19104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:44:50.342097 19098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:44:50.342753 19098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:44:50.343925 19098 hybrid_clock.cc:648] HybridClock initialized: now 1790513090343911 us; error 25 us; skew 500 ppm
I20260927 12:44:50.345899 19098 webserver.cc:460] Webserver started at http://127.0.98.1:43111/ using document root <none> and password file <none>
I20260927 12:44:50.346450 19098 fs_manager.cc:359] Metadata directory not provided
I20260927 12:44:50.346511 19098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:44:50.346673 19098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:44:50.348728 19098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512840836255-392-0/minicluster-data/ts-0/data/instance:
uuid: "08f3a2e10361467eb16c8716f7710f78"
format_stamp: "Formatted at 2026-09-27 12:44:50 on dist-test-slave-bnc1"
server_key: "a270da49e2245b7dab6ee09a724e0b9c"
server_key_iv: "f42ad67188bfe14215fad583c27369b1"
server_key_version: "encryptionkey@0"
I20260927 12:44:50.349244 19098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512840836255-392-0/minicluster-data/ts-0/wal/instance:
uuid: "08f3a2e10361467eb16c8716f7710f78"
format_stamp: "Formatted at 2026-09-27 12:44:50 on dist-test-slave-bnc1"
server_key: "a270da49e2245b7dab6ee09a724e0b9c"
server_key_iv: "f42ad67188bfe14215fad583c27369b1"
server_key_version: "encryptionkey@0"
I20260927 12:44:50.352869 19098 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 12:44:50.355283 19112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:44:50.356496 19098 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 12:44:50.356587 19098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512840836255-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512840836255-392-0/minicluster-data/ts-0/wal
uuid: "08f3a2e10361467eb16c8716f7710f78"
format_stamp: "Formatted at 2026-09-27 12:44:50 on dist-test-slave-bnc1"
server_key: "a270da49e2245b7dab6ee09a724e0b9c"
server_key_iv: "f42ad67188bfe14215fad583c27369b1"
server_key_version: "encryptionkey@0"
I20260927 12:44:50.356669 19098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512840836255-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512840836255-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512840836255-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:44:50.379151 19098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:44:50.379829 19098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:44:50.379981 19098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:44:50.380527 19098 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:44:50.381523 19098 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:44:50.381628 19098 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:44:50.381709 19098 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:44:50.381752 19098 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:44:50.391029 19098 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42157
I20260927 12:44:50.391047 19225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42157 every 8 connection(s)
I20260927 12:44:50.391994 19098 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:44:50.394472 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 19098
I20260927 12:44:50.394629 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512840836255-392-0/minicluster-data/ts-0/wal/instance
I20260927 12:44:50.394974 392 external_mini_cluster.cc:1428] Setting key 885af063c80e71578144cab0586421b6
I20260927 12:44:50.397380 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512840836255-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512840836255-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512840836255-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35335
--builtin_ntp_servers=127.0.98.20:39865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 12:44:50.403275 19077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:50.393854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60181 (local address 127.0.98.62:35335)
0927 12:44:50.393994 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:50.393997 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:50.395311 (+ 1314us) server_negotiation.cc:407] Connection header received
0927 12:44:50.397099 (+ 1788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:50.397102 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:50.397150 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:50.397259 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:44:50.398654 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:50.399603 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:50.400793 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:50.401243 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:50.401378 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:50.402572 (+ 1194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:44:50.402580 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:44:50.402585 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:44:50.402604 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:44:50.402630 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:44:50.402645 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:44:50.402742 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:50.402949 (+ 207us) server_negotiation.cc:299] Negotiation successful
0927 12:44:50.403036 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 12:44:50.404129 19228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:50.394283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35335 (local address 127.0.98.1:60181)
0927 12:44:50.395116 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:50.395176 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 12:44:50.396830 (+ 1654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:44:50.397499 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:44:50.397510 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:44:50.397708 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:44:50.398476 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:50.398499 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:50.399754 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:50.399757 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:50.400653 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:50.400662 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:50.401406 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:50.401409 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:50.401618 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:50.402290 (+ 672us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 12:44:50.402292 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:44:50.402336 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 12:44:50.402401 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:44:50.402766 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:44:50.402775 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:44:50.402780 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 12:44:50.402918 (+ 138us) client_negotiation.cc:240] Negotiation successful
0927 12:44:50.403221 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":111,"threads_started":1}
I20260927 12:44:50.405596 19226 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35335
I20260927 12:44:50.405881 19226 heartbeater.cc:464] Registering TS with master...
I20260927 12:44:50.406606 19226 heartbeater.cc:511] Master 127.0.98.62:35335 requested a full tablet report, sending...
I20260927 12:44:50.408283 19038 ts_manager.cc:194] Registered new tserver with Master: 08f3a2e10361467eb16c8716f7710f78 (127.0.98.1:42157)
I20260927 12:44:50.409437 19038 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:60181
W20260927 12:44:50.509083 19229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:44:50.509373 19229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:44:50.509414 19229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:44:50.513550 19229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:44:50.513688 19229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:44:50.513782 19229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:44:50.518939 19229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512840836255-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512840836255-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512840836255-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:44:50.520108 19229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:44:50.521639 19229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:44:50.531368 19235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:44:50.531426 19229 server_base.cc:1034] running on GCE node
W20260927 12:44:50.531376 19234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:44:50.531378 19237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:44:50.531956 19229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:44:50.532648 19229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:44:50.533809 19229 hybrid_clock.cc:648] HybridClock initialized: now 1790513090533792 us; error 28 us; skew 500 ppm
I20260927 12:44:50.535725 19229 webserver.cc:460] Webserver started at http://127.0.98.2:38559/ using document root <none> and password file <none>
I20260927 12:44:50.536309 19229 fs_manager.cc:359] Metadata directory not provided
I20260927 12:44:50.536372 19229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:44:50.536610 19229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:44:50.538810 19229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512840836255-392-0/minicluster-data/ts-1/data/instance:
uuid: "4426197c8af44982baf64877611a2781"
format_stamp: "Formatted at 2026-09-27 12:44:50 on dist-test-slave-bnc1"
server_key: "db43ea6305ae6ab2730c1ed214e85797"
server_key_iv: "eacdd6931b31120a58916a7935dfa462"
server_key_version: "encryptionkey@0"
I20260927 12:44:50.539390 19229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512840836255-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4426197c8af44982baf64877611a2781"
format_stamp: "Formatted at 2026-09-27 12:44:50 on dist-test-slave-bnc1"
server_key: "db43ea6305ae6ab2730c1ed214e85797"
server_key_iv: "eacdd6931b31120a58916a7935dfa462"
server_key_version: "encryptionkey@0"
I20260927 12:44:50.543081 19229 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260927 12:44:50.545514 19243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:44:50.546710 19229 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:44:50.546801 19229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512840836255-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512840836255-392-0/minicluster-data/ts-1/wal
uuid: "4426197c8af44982baf64877611a2781"
format_stamp: "Formatted at 2026-09-27 12:44:50 on dist-test-slave-bnc1"
server_key: "db43ea6305ae6ab2730c1ed214e85797"
server_key_iv: "eacdd6931b31120a58916a7935dfa462"
server_key_version: "encryptionkey@0"
I20260927 12:44:50.546885 19229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512840836255-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512840836255-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512840836255-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:44:50.561131 19229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:44:50.561872 19229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:44:50.562016 19229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:44:50.562604 19229 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:44:50.563541 19229 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:44:50.563589 19229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:44:50.563658 19229 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:44:50.563694 19229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:44:50.572062 19229 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41249
I20260927 12:44:50.572221 19356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41249 every 8 connection(s)
I20260927 12:44:50.573056 19229 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:44:50.574031 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 19229
I20260927 12:44:50.574149 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512840836255-392-0/minicluster-data/ts-1/wal/instance
I20260927 12:44:50.574441 392 external_mini_cluster.cc:1428] Setting key f169c0492f844098592634f83ec27dbd
I20260927 12:44:50.576678 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512840836255-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512840836255-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512840836255-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35335
--builtin_ntp_servers=127.0.98.20:39865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 12:44:50.582796 19077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:50.575147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40545 (local address 127.0.98.62:35335)
0927 12:44:50.575308 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:50.575312 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:50.576480 (+ 1168us) server_negotiation.cc:407] Connection header received
0927 12:44:50.578003 (+ 1523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:50.578006 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:50.578062 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:50.578171 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:44:50.579055 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:50.579574 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:50.580771 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:50.581009 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:50.581121 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:50.582302 (+ 1181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:44:50.582305 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:44:50.582307 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:44:50.582319 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:44:50.582340 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:44:50.582347 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:44:50.582423 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:50.582583 (+ 160us) server_negotiation.cc:299] Negotiation successful
0927 12:44:50.582644 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 12:44:50.583602 19359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:50.575601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35335 (local address 127.0.98.2:40545)
0927 12:44:50.576342 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:50.576383 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 12:44:50.577806 (+ 1423us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:44:50.578349 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:44:50.578356 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:44:50.578437 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:44:50.578916 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:50.578927 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:50.579707 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:50.579711 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:50.580648 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:50.580658 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:50.581162 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:50.581166 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:50.581334 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:50.582015 (+ 681us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 12:44:50.582016 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:44:50.582057 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 12:44:50.582119 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:44:50.582438 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:44:50.582449 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:44:50.582458 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 12:44:50.582582 (+ 124us) client_negotiation.cc:240] Negotiation successful
0927 12:44:50.582855 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":143,"threads_started":1}
I20260927 12:44:50.584843 19357 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35335
I20260927 12:44:50.585073 19357 heartbeater.cc:464] Registering TS with master...
I20260927 12:44:50.585638 19357 heartbeater.cc:511] Master 127.0.98.62:35335 requested a full tablet report, sending...
I20260927 12:44:50.586745 19038 ts_manager.cc:194] Registered new tserver with Master: 4426197c8af44982baf64877611a2781 (127.0.98.2:41249)
I20260927 12:44:50.587304 19038 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:40545
W20260927 12:44:50.692128 19360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:44:50.692478 19360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:44:50.692562 19360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:44:50.696734 19360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:44:50.696800 19360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:44:50.696882 19360 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:44:50.701774 19360 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512840836255-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512840836255-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19360
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512840836255-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:44:50.702804 19360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:44:50.704145 19360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:44:50.713959 19365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:44:50.714023 19366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:44:50.713959 19368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:44:50.714423 19360 server_base.cc:1034] running on GCE node
I20260927 12:44:50.714854 19360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:44:50.715543 19360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:44:50.716737 19360 hybrid_clock.cc:648] HybridClock initialized: now 1790513090716722 us; error 27 us; skew 500 ppm
I20260927 12:44:50.718604 19360 webserver.cc:460] Webserver started at http://127.0.98.3:40515/ using document root <none> and password file <none>
I20260927 12:44:50.719182 19360 fs_manager.cc:359] Metadata directory not provided
I20260927 12:44:50.719254 19360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:44:50.719487 19360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:44:50.721508 19360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512840836255-392-0/minicluster-data/ts-2/data/instance:
uuid: "4be7338de151428b936ee6fe2aa78195"
format_stamp: "Formatted at 2026-09-27 12:44:50 on dist-test-slave-bnc1"
server_key: "c8790e7fb0771efed3a530ba39df79bd"
server_key_iv: "18ec9aee339bf331d93ad31b3464a8ce"
server_key_version: "encryptionkey@0"
I20260927 12:44:50.722133 19360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512840836255-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4be7338de151428b936ee6fe2aa78195"
format_stamp: "Formatted at 2026-09-27 12:44:50 on dist-test-slave-bnc1"
server_key: "c8790e7fb0771efed3a530ba39df79bd"
server_key_iv: "18ec9aee339bf331d93ad31b3464a8ce"
server_key_version: "encryptionkey@0"
I20260927 12:44:50.725775 19360 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:44:50.728193 19374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:44:50.729398 19360 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:44:50.729516 19360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512840836255-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512840836255-392-0/minicluster-data/ts-2/wal
uuid: "4be7338de151428b936ee6fe2aa78195"
format_stamp: "Formatted at 2026-09-27 12:44:50 on dist-test-slave-bnc1"
server_key: "c8790e7fb0771efed3a530ba39df79bd"
server_key_iv: "18ec9aee339bf331d93ad31b3464a8ce"
server_key_version: "encryptionkey@0"
I20260927 12:44:50.729676 19360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512840836255-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512840836255-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512840836255-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:44:50.743386 19360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:44:50.744040 19360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:44:50.744232 19360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:44:50.744796 19360 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:44:50.745781 19360 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:44:50.745831 19360 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:44:50.745898 19360 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:44:50.745935 19360 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:44:50.755075 19360 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39089
I20260927 12:44:50.755097 19487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39089 every 8 connection(s)
I20260927 12:44:50.756027 19360 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:44:50.762719 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 19360
I20260927 12:44:50.762858 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512840836255-392-0/minicluster-data/ts-2/wal/instance
I20260927 12:44:50.763134 392 external_mini_cluster.cc:1428] Setting key e25324559a5d34d4f98f1a9013f55397
I20260927 12:44:50.766476 19077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:50.757962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55023 (local address 127.0.98.62:35335)
0927 12:44:50.758108 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:50.758111 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:50.759314 (+ 1203us) server_negotiation.cc:407] Connection header received
0927 12:44:50.760876 (+ 1562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:50.760879 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:50.760983 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:50.761063 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:44:50.762134 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:50.762676 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:50.764291 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:50.764595 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:50.764712 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:50.765923 (+ 1211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:44:50.765927 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:44:50.765929 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:44:50.765941 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:44:50.765959 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:44:50.765966 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:44:50.766041 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:50.766266 (+ 225us) server_negotiation.cc:299] Negotiation successful
0927 12:44:50.766335 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 12:44:50.767364 19490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:50.758411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35335 (local address 127.0.98.3:55023)
0927 12:44:50.759125 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:50.759169 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 12:44:50.760650 (+ 1481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:44:50.761263 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:44:50.761270 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:44:50.761359 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:44:50.761992 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:50.762005 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:50.762868 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:50.762872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:50.764154 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:50.764165 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:50.764757 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:50.764761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:50.764938 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:50.765655 (+ 717us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 12:44:50.765657 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:44:50.765704 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 12:44:50.765768 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:44:50.766055 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:44:50.766067 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:44:50.766076 (+ 9us) client_negotiation.cc:769] Sending connection context
0927 12:44:50.766265 (+ 189us) client_negotiation.cc:240] Negotiation successful
0927 12:44:50.766579 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":109,"threads_started":1}
I20260927 12:44:50.768430 19488 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35335
I20260927 12:44:50.768636 19488 heartbeater.cc:464] Registering TS with master...
I20260927 12:44:50.769105 19488 heartbeater.cc:511] Master 127.0.98.62:35335 requested a full tablet report, sending...
I20260927 12:44:50.770092 19038 ts_manager.cc:194] Registered new tserver with Master: 4be7338de151428b936ee6fe2aa78195 (127.0.98.3:39089)
I20260927 12:44:50.770695 19038 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:55023
I20260927 12:44:50.777146 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 12:44:50.783542 19077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:50.779159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41660 (local address 127.0.98.62:35335)
0927 12:44:50.779351 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:50.779356 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:50.779510 (+ 154us) server_negotiation.cc:407] Connection header received
0927 12:44:50.779654 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:50.779658 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:50.779708 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:50.779796 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:44:50.780079 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:50.780591 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:50.781721 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:50.781947 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:50.782065 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:50.783034 (+ 969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:44:50.783039 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:44:50.783047 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:44:50.783058 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:44:50.783077 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:44:50.783086 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:44:50.783226 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:50.783354 (+ 128us) server_negotiation.cc:299] Negotiation successful
0927 12:44:50.783395 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260927 12:44:50.786671 19038 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41660:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 12:44:50.789726 19038 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:44:50.804905 19499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:50.798537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42157 (local address 127.0.0.1:55748)
0927 12:44:50.799515 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:50.799572 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 12:44:50.799704 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:44:50.800510 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:44:50.800515 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:44:50.800537 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:44:50.800673 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:50.800680 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:50.802298 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:50.802302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:50.803463 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:50.803472 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:50.804461 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:50.804465 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:50.804575 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:50.804596 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 12:44:50.804649 (+ 53us) client_negotiation.cc:240] Negotiation successful
0927 12:44:50.804718 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":799,"mutex_wait_us":33,"thread_start_us":97,"threads_started":1}
I20260927 12:44:50.804942 19504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:50.799433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59848 (local address 127.0.98.3:39089)
0927 12:44:50.800020 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:50.800034 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:50.800060 (+ 26us) server_negotiation.cc:407] Connection header received
0927 12:44:50.800156 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:50.800162 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:50.800364 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:50.800519 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:44:50.800961 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:50.802169 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:50.803521 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:50.804321 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:50.804437 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:50.804621 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:50.804672 (+ 51us) server_negotiation.cc:299] Negotiation successful
0927 12:44:50.804782 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":190,"threads_started":1}
I20260927 12:44:50.805168 19503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:50.799199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46122 (local address 127.0.98.2:41249)
0927 12:44:50.799751 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:50.799764 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:50.800074 (+ 310us) server_negotiation.cc:407] Connection header received
0927 12:44:50.800139 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:50.800143 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:50.800336 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:50.800489 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:44:50.800881 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:50.801751 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:50.803334 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:50.803975 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:50.804148 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:50.804359 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:50.804842 (+ 483us) server_negotiation.cc:299] Negotiation successful
0927 12:44:50.804968 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":96,"threads_started":1}
I20260927 12:44:50.804762 19502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:50.799003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41249 (local address 127.0.0.1:46122)
0927 12:44:50.799890 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:50.799902 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 12:44:50.799987 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:44:50.800630 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:44:50.800633 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:44:50.800648 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:44:50.800738 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:50.800746 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:50.801876 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:50.801882 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:50.803177 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:50.803189 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:50.804146 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:50.804150 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:50.804260 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:50.804300 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 12:44:50.804442 (+ 142us) client_negotiation.cc:240] Negotiation successful
0927 12:44:50.804566 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":825,"thread_start_us":108,"threads_started":1}
I20260927 12:44:50.805397 19500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:50.798724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39089 (local address 127.0.0.1:59848)
0927 12:44:50.799515 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:50.799572 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 12:44:50.799702 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:44:50.800679 (+ 977us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:44:50.800682 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:44:50.800698 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:44:50.800787 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:50.800816 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:50.802337 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:50.802340 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:50.803402 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:50.803411 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:50.804442 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:50.804446 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:50.804543 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:50.804566 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 12:44:50.804634 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 12:44:50.804694 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":652,"mutex_wait_us":23,"thread_start_us":102,"threads_started":1}
I20260927 12:44:50.805666 19501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:50.798702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55748 (local address 127.0.98.1:42157)
0927 12:44:50.799998 (+ 1296us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:50.800012 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:50.800040 (+ 28us) server_negotiation.cc:407] Connection header received
0927 12:44:50.800122 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:50.800127 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:50.800336 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:50.800843 (+ 507us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:44:50.801014 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:50.802169 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:50.803577 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:50.804321 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:50.804866 (+ 545us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:50.805291 (+ 425us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:50.805341 (+ 50us) server_negotiation.cc:299] Negotiation successful
0927 12:44:50.805478 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1194,"thread_start_us":111,"threads_started":1}
I20260927 12:44:50.808216 19291 tablet_service.cc:1467] Processing CreateTablet for tablet 0b01206bfc8c4e2eaaaaa512ebf4785b (DEFAULT_TABLE table=test-table [id=fb9ef00db27d4a7a963a47881556bac7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:44:50.808535 19422 tablet_service.cc:1467] Processing CreateTablet for tablet 0b01206bfc8c4e2eaaaaa512ebf4785b (DEFAULT_TABLE table=test-table [id=fb9ef00db27d4a7a963a47881556bac7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:44:50.808535 19160 tablet_service.cc:1467] Processing CreateTablet for tablet 0b01206bfc8c4e2eaaaaa512ebf4785b (DEFAULT_TABLE table=test-table [id=fb9ef00db27d4a7a963a47881556bac7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:44:50.809423 19291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b01206bfc8c4e2eaaaaa512ebf4785b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:44:50.809567 19160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b01206bfc8c4e2eaaaaa512ebf4785b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:44:50.809567 19422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b01206bfc8c4e2eaaaaa512ebf4785b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:44:50.815938 19505 tablet_bootstrap.cc:492] T 0b01206bfc8c4e2eaaaaa512ebf4785b P 4426197c8af44982baf64877611a2781: Bootstrap starting.
I20260927 12:44:50.817473 19507 tablet_bootstrap.cc:492] T 0b01206bfc8c4e2eaaaaa512ebf4785b P 4be7338de151428b936ee6fe2aa78195: Bootstrap starting.
I20260927 12:44:50.817838 19506 tablet_bootstrap.cc:492] T 0b01206bfc8c4e2eaaaaa512ebf4785b P 08f3a2e10361467eb16c8716f7710f78: Bootstrap starting.
I20260927 12:44:50.818521 19505 tablet_bootstrap.cc:654] T 0b01206bfc8c4e2eaaaaa512ebf4785b P 4426197c8af44982baf64877611a2781: Neither blocks nor log segments found. Creating new log.
I20260927 12:44:50.819391 19505 log.cc:824] T 0b01206bfc8c4e2eaaaaa512ebf4785b P 4426197c8af44982baf64877611a2781: Log is configured to *not* fsync() on all Append() calls
I20260927 12:44:50.819629 19507 tablet_bootstrap.cc:654] T 0b01206bfc8c4e2eaaaaa512ebf4785b P 4be7338de151428b936ee6fe2aa78195: Neither blocks nor log segments found. Creating new log.
I20260927 12:44:50.819713 19506 tablet_bootstrap.cc:654] T 0b01206bfc8c4e2eaaaaa512ebf4785b P 08f3a2e10361467eb16c8716f7710f78: Neither blocks nor log segments found. Creating new log.
I20260927 12:44:50.820412 19507 log.cc:824] T 0b01206bfc8c4e2eaaaaa512ebf4785b P 4be7338de151428b936ee6fe2aa78195: Log is configured to *not* fsync() on all Append() calls
I20260927 12:44:50.820568 19506 log.cc:824] T 0b01206bfc8c4e2eaaaaa512ebf4785b P 08f3a2e10361467eb16c8716f7710f78: Log is configured to *not* fsync() on all Append() calls
I20260927 12:44:50.821795 19505 tablet_bootstrap.cc:492] T 0b01206bfc8c4e2eaaaaa512ebf4785b P 4426197c8af44982baf64877611a2781: No bootstrap required, opened a new log
I20260927 12:44:50.822008 19505 ts_tablet_manager.cc:1397] T 0b01206bfc8c4e2eaaaaa512ebf4785b P 4426197c8af44982baf64877611a2781: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260927 12:44:50.822376 19507 tablet_bootstrap.cc:492] T 0b01206bfc8c4e2eaaaaa512ebf4785b P 4be7338de151428b936ee6fe2aa78195: No bootstrap required, opened a new log
I20260927 12:44:50.822399 19506 tablet_bootstrap.cc:492] T 0b01206bfc8c4e2eaaaaa512ebf4785b P 08f3a2e10361467eb16c8716f7710f78: No bootstrap required, opened a new log
I20260927 12:44:50.822558 19506 ts_tablet_manager.cc:1397] T 0b01206bfc8c4e2eaaaaa512ebf4785b P 08f3a2e10361467eb16c8716f7710f78: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 12:44:50.822558 19507 ts_tablet_manager.cc:1397] T 0b01206bfc8c4e2eaaaaa512ebf4785b P 4be7338de151428b936ee6fe2aa78195: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 12:44:50.825385 19505 raft_consensus.cc:348] T 0b01206bfc8c4e2eaaaaa512ebf4785b P 4426197c8af44982baf64877611a2781 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08f3a2e10361467eb16c8716f7710f78" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42157 } } peers { permanent_uuid: "4be7338de151428b936ee6fe2aa78195" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39089 } } peers { permanent_uuid: "4426197c8af44982baf64877611a2781" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41249 } }
I20260927 12:44:50.825726 19505 raft_consensus.cc:374] T 0b01206bfc8c4e2eaaaaa512ebf4785b P 4426197c8af44982baf64877611a2781 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:44:50.825790 19505 raft_consensus.cc:729] T 0b01206bfc8c4e2eaaaaa512ebf4785b P 4426197c8af44982baf64877611a2781 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4426197c8af44982baf64877611a2781, State: Initialized, Role: FOLLOWER
I20260927 12:44:50.826249 19505 consensus_queue.cc:260] T 0b01206bfc8c4e2eaaaaa512ebf4785b P 4426197c8af44982baf64877611a2781 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08f3a2e10361467eb16c8716f7710f78" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42157 } } peers { permanent_uuid: "4be7338de151428b936ee6fe2aa78195" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39089 } } peers { permanent_uuid: "4426197c8af44982baf64877611a2781" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41249 } }
I20260927 12:44:50.826886 19357 heartbeater.cc:503] Master 127.0.98.62:35335 was elected leader, sending a full tablet report...
I20260927 12:44:50.826846 19506 raft_consensus.cc:348] T 0b01206bfc8c4e2eaaaaa512ebf4785b P 08f3a2e10361467eb16c8716f7710f78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08f3a2e10361467eb16c8716f7710f78" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42157 } } peers { permanent_uuid: "4be7338de151428b936ee6fe2aa78195" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39089 } } peers { permanent_uuid: "4426197c8af44982baf64877611a2781" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41249 } }
I20260927 12:44:50.827109 19506 raft_consensus.cc:374] T 0b01206bfc8c4e2eaaaaa512ebf4785b P 08f3a2e10361467eb16c8716f7710f78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:44:50.827169 19506 raft_consensus.cc:729] T 0b01206bfc8c4e2eaaaaa512ebf4785b P 08f3a2e10361467eb16c8716f7710f78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08f3a2e10361467eb16c8716f7710f78, State: Initialized, Role: FOLLOWER
I20260927 12:44:50.827241 19505 ts_tablet_manager.cc:1428] T 0b01206bfc8c4e2eaaaaa512ebf4785b P 4426197c8af44982baf64877611a2781: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260927 12:44:50.827597 19506 consensus_queue.cc:260] T 0b01206bfc8c4e2eaaaaa512ebf4785b P 08f3a2e10361467eb16c8716f7710f78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08f3a2e10361467eb16c8716f7710f78" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42157 } } peers { permanent_uuid: "4be7338de151428b936ee6fe2aa78195" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39089 } } peers { permanent_uuid: "4426197c8af44982baf64877611a2781" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41249 } }
I20260927 12:44:50.826846 19507 raft_consensus.cc:348] T 0b01206bfc8c4e2eaaaaa512ebf4785b P 4be7338de151428b936ee6fe2aa78195 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08f3a2e10361467eb16c8716f7710f78" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42157 } } peers { permanent_uuid: "4be7338de151428b936ee6fe2aa78195" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39089 } } peers { permanent_uuid: "4426197c8af44982baf64877611a2781" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41249 } }
I20260927 12:44:50.827834 19507 raft_consensus.cc:374] T 0b01206bfc8c4e2eaaaaa512ebf4785b P 4be7338de151428b936ee6fe2aa78195 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:44:50.827909 19507 raft_consensus.cc:729] T 0b01206bfc8c4e2eaaaaa512ebf4785b P 4be7338de151428b936ee6fe2aa78195 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4be7338de151428b936ee6fe2aa78195, State: Initialized, Role: FOLLOWER
I20260927 12:44:50.828619 19506 ts_tablet_manager.cc:1428] T 0b01206bfc8c4e2eaaaaa512ebf4785b P 08f3a2e10361467eb16c8716f7710f78: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 12:44:50.828401 19507 consensus_queue.cc:260] T 0b01206bfc8c4e2eaaaaa512ebf4785b P 4be7338de151428b936ee6fe2aa78195 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08f3a2e10361467eb16c8716f7710f78" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42157 } } peers { permanent_uuid: "4be7338de151428b936ee6fe2aa78195" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39089 } } peers { permanent_uuid: "4426197c8af44982baf64877611a2781" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41249 } }
I20260927 12:44:50.828634 19226 heartbeater.cc:503] Master 127.0.98.62:35335 was elected leader, sending a full tablet report...
I20260927 12:44:50.829198 19488 heartbeater.cc:503] Master 127.0.98.62:35335 was elected leader, sending a full tablet report...
I20260927 12:44:50.830101 19507 ts_tablet_manager.cc:1428] T 0b01206bfc8c4e2eaaaaa512ebf4785b P 4be7338de151428b936ee6fe2aa78195: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260927 12:44:50.858630 19512 raft_consensus.cc:482] T 0b01206bfc8c4e2eaaaaa512ebf4785b P 08f3a2e10361467eb16c8716f7710f78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:44:50.858880 19512 raft_consensus.cc:504] T 0b01206bfc8c4e2eaaaaa512ebf4785b P 08f3a2e10361467eb16c8716f7710f78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08f3a2e10361467eb16c8716f7710f78" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42157 } } peers { permanent_uuid: "4be7338de151428b936ee6fe2aa78195" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39089 } } peers { permanent_uuid: "4426197c8af44982baf64877611a2781" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41249 } }
I20260927 12:44:50.860239 19512 leader_election.cc:288] T 0b01206bfc8c4e2eaaaaa512ebf4785b P 08f3a2e10361467eb16c8716f7710f78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4be7338de151428b936ee6fe2aa78195 (127.0.98.3:39089), 4426197c8af44982baf64877611a2781 (127.0.98.2:41249)
I20260927 12:44:50.864820 19503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:50.860807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43993 (local address 127.0.98.2:41249)
0927 12:44:50.860968 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:50.860973 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:50.861018 (+ 45us) server_negotiation.cc:407] Connection header received
0927 12:44:50.861115 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:50.861117 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:50.861155 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:50.861250 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:44:50.861585 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:50.862250 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:50.863616 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:50.864181 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:50.864277 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:50.864334 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:50.864621 (+ 287us) server_negotiation.cc:299] Negotiation successful
0927 12:44:50.864705 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 12:44:50.864820 19504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:50.860505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50889 (local address 127.0.98.3:39089)
0927 12:44:50.860660 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:50.860665 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:50.860682 (+ 17us) server_negotiation.cc:407] Connection header received
0927 12:44:50.860761 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:50.860767 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:50.860815 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:50.860931 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:44:50.861240 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:50.862010 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:50.863537 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:50.864148 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:50.864273 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:50.864334 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:50.864621 (+ 287us) server_negotiation.cc:299] Negotiation successful
0927 12:44:50.864692 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 12:44:50.864876 19228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:50.860412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39089 (local address 127.0.98.1:50889)
0927 12:44:50.860542 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:50.860558 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 12:44:50.860651 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:44:50.860944 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:44:50.860947 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:44:50.860969 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:44:50.861105 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:50.861116 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:50.862123 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:50.862126 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:50.863426 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:50.863437 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:50.864290 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:50.864294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:50.864476 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:50.864517 (+ 41us) client_negotiation.cc:769] Sending connection context
0927 12:44:50.864593 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 12:44:50.864719 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":92}
I20260927 12:44:50.865072 19514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:50.860499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41249 (local address 127.0.98.1:43993)
0927 12:44:50.860947 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:50.860960 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:44:50.861037 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:44:50.861321 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:44:50.861324 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:44:50.861341 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:44:50.861451 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:50.861456 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:50.862367 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:50.862371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:50.863500 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:50.863509 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:50.864290 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:50.864294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:50.864471 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:50.864514 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 12:44:50.864593 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 12:44:50.864719 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"mutex_wait_us":73,"thread_start_us":119,"threads_started":1}
I20260927 12:44:50.865700 19311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b01206bfc8c4e2eaaaaa512ebf4785b" candidate_uuid: "08f3a2e10361467eb16c8716f7710f78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4426197c8af44982baf64877611a2781" is_pre_election: true
I20260927 12:44:50.865671 19442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b01206bfc8c4e2eaaaaa512ebf4785b" candidate_uuid: "08f3a2e10361467eb16c8716f7710f78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4be7338de151428b936ee6fe2aa78195" is_pre_election: true
I20260927 12:44:50.866007 19442 raft_consensus.cc:2454] T 0b01206bfc8c4e2eaaaaa512ebf4785b P 4be7338de151428b936ee6fe2aa78195 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08f3a2e10361467eb16c8716f7710f78 in term 0.
I20260927 12:44:50.866007 19311 raft_consensus.cc:2454] T 0b01206bfc8c4e2eaaaaa512ebf4785b P 4426197c8af44982baf64877611a2781 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08f3a2e10361467eb16c8716f7710f78 in term 0.
I20260927 12:44:50.866555 19114 leader_election.cc:302] T 0b01206bfc8c4e2eaaaaa512ebf4785b P 08f3a2e10361467eb16c8716f7710f78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 08f3a2e10361467eb16c8716f7710f78, 4426197c8af44982baf64877611a2781, 4be7338de151428b936ee6fe2aa78195; no voters:
I20260927 12:44:50.866837 19512 raft_consensus.cc:2781] T 0b01206bfc8c4e2eaaaaa512ebf4785b P 08f3a2e10361467eb16c8716f7710f78 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:44:50.866918 19512 raft_consensus.cc:482] T 0b01206bfc8c4e2eaaaaa512ebf4785b P 08f3a2e10361467eb16c8716f7710f78 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:44:50.866948 19512 raft_consensus.cc:3037] T 0b01206bfc8c4e2eaaaaa512ebf4785b P 08f3a2e10361467eb16c8716f7710f78 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:44:50.868013 19512 raft_consensus.cc:504] T 0b01206bfc8c4e2eaaaaa512ebf4785b P 08f3a2e10361467eb16c8716f7710f78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08f3a2e10361467eb16c8716f7710f78" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42157 } } peers { permanent_uuid: "4be7338de151428b936ee6fe2aa78195" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39089 } } peers { permanent_uuid: "4426197c8af44982baf64877611a2781" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41249 } }
I20260927 12:44:50.868386 19512 leader_election.cc:288] T 0b01206bfc8c4e2eaaaaa512ebf4785b P 08f3a2e10361467eb16c8716f7710f78 [CANDIDATE]: Term 1 election: Requested vote from peers 4be7338de151428b936ee6fe2aa78195 (127.0.98.3:39089), 4426197c8af44982baf64877611a2781 (127.0.98.2:41249)
I20260927 12:44:50.868727 19442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b01206bfc8c4e2eaaaaa512ebf4785b" candidate_uuid: "08f3a2e10361467eb16c8716f7710f78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4be7338de151428b936ee6fe2aa78195"
I20260927 12:44:50.868728 19311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b01206bfc8c4e2eaaaaa512ebf4785b" candidate_uuid: "08f3a2e10361467eb16c8716f7710f78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4426197c8af44982baf64877611a2781"
I20260927 12:44:50.868866 19311 raft_consensus.cc:3037] T 0b01206bfc8c4e2eaaaaa512ebf4785b P 4426197c8af44982baf64877611a2781 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:44:50.868865 19442 raft_consensus.cc:3037] T 0b01206bfc8c4e2eaaaaa512ebf4785b P 4be7338de151428b936ee6fe2aa78195 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:44:50.869964 19311 raft_consensus.cc:2454] T 0b01206bfc8c4e2eaaaaa512ebf4785b P 4426197c8af44982baf64877611a2781 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08f3a2e10361467eb16c8716f7710f78 in term 1.
I20260927 12:44:50.870019 19442 raft_consensus.cc:2454] T 0b01206bfc8c4e2eaaaaa512ebf4785b P 4be7338de151428b936ee6fe2aa78195 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08f3a2e10361467eb16c8716f7710f78 in term 1.
I20260927 12:44:50.870324 19115 leader_election.cc:302] T 0b01206bfc8c4e2eaaaaa512ebf4785b P 08f3a2e10361467eb16c8716f7710f78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08f3a2e10361467eb16c8716f7710f78, 4426197c8af44982baf64877611a2781; no voters:
I20260927 12:44:50.870507 19512 raft_consensus.cc:2781] T 0b01206bfc8c4e2eaaaaa512ebf4785b P 08f3a2e10361467eb16c8716f7710f78 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:44:50.870723 19512 raft_consensus.cc:686] T 0b01206bfc8c4e2eaaaaa512ebf4785b P 08f3a2e10361467eb16c8716f7710f78 [term 1 LEADER]: Becoming Leader. State: Replica: 08f3a2e10361467eb16c8716f7710f78, State: Running, Role: LEADER
I20260927 12:44:50.871042 19512 consensus_queue.cc:237] T 0b01206bfc8c4e2eaaaaa512ebf4785b P 08f3a2e10361467eb16c8716f7710f78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08f3a2e10361467eb16c8716f7710f78" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42157 } } peers { permanent_uuid: "4be7338de151428b936ee6fe2aa78195" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39089 } } peers { permanent_uuid: "4426197c8af44982baf64877611a2781" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41249 } }
I20260927 12:44:50.874452 19038 catalog_manager.cc:5495] T 0b01206bfc8c4e2eaaaaa512ebf4785b P 08f3a2e10361467eb16c8716f7710f78 reported cstate change: term changed from 0 to 1, leader changed from <none> to 08f3a2e10361467eb16c8716f7710f78 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "08f3a2e10361467eb16c8716f7710f78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08f3a2e10361467eb16c8716f7710f78" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4be7338de151428b936ee6fe2aa78195" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4426197c8af44982baf64877611a2781" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41249 } health_report { overall_health: UNKNOWN } } }
W20260927 12:44:50.893198 19227 tablet.cc:2369] T 0b01206bfc8c4e2eaaaaa512ebf4785b P 08f3a2e10361467eb16c8716f7710f78: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:44:50.903059 19501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:50.899306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55760 (local address 127.0.98.1:42157)
0927 12:44:50.899449 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:50.899456 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:50.899472 (+ 16us) server_negotiation.cc:407] Connection header received
0927 12:44:50.899667 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:50.899669 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:50.899727 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:50.899840 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:44:50.900086 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:50.900595 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:50.901519 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:50.901803 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:50.901916 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:50.902225 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:44:50.902624 (+ 399us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:44:50.902799 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:50.902861 (+ 62us) server_negotiation.cc:299] Negotiation successful
0927 12:44:50.902897 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 12:44:50.909260 19442 raft_consensus.cc:1261] T 0b01206bfc8c4e2eaaaaa512ebf4785b P 4be7338de151428b936ee6fe2aa78195 [term 1 FOLLOWER]: Refusing update from remote peer 08f3a2e10361467eb16c8716f7710f78: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:44:50.909297 19311 raft_consensus.cc:1261] T 0b01206bfc8c4e2eaaaaa512ebf4785b P 4426197c8af44982baf64877611a2781 [term 1 FOLLOWER]: Refusing update from remote peer 08f3a2e10361467eb16c8716f7710f78: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:44:50.910104 19515 consensus_queue.cc:1035] T 0b01206bfc8c4e2eaaaaa512ebf4785b P 08f3a2e10361467eb16c8716f7710f78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4426197c8af44982baf64877611a2781" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:44:50.910395 19512 consensus_queue.cc:1035] T 0b01206bfc8c4e2eaaaaa512ebf4785b P 08f3a2e10361467eb16c8716f7710f78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4be7338de151428b936ee6fe2aa78195" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:44:50.920214 19519 mvcc.cc:205] Tried to move back new op lower bound from 7333941620356112384 to 7333941620210601984. Current Snapshot: MvccSnapshot[applied={T|T < 7333941620356112384}]
I20260927 12:44:50.928298 19038 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41660:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:44:50.928488 19038 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41660:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:44:50.930998 19038 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3863be4dc4774eb3a2f220a905630c92: Sending DeleteTablet for 3 replicas of tablet 0b01206bfc8c4e2eaaaaa512ebf4785b
I20260927 12:44:50.931780 19160 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b01206bfc8c4e2eaaaaa512ebf4785b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:44:50 UTC) from {username='slave'} at 127.0.0.1:55748
I20260927 12:44:50.931821 19291 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b01206bfc8c4e2eaaaaa512ebf4785b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:44:50 UTC) from {username='slave'} at 127.0.0.1:46122
I20260927 12:44:50.931859 19422 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b01206bfc8c4e2eaaaaa512ebf4785b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:44:50 UTC) from {username='slave'} at 127.0.0.1:59848
I20260927 12:44:50.932265 19529 tablet_replica.cc:330] T 0b01206bfc8c4e2eaaaaa512ebf4785b P 08f3a2e10361467eb16c8716f7710f78: stopping tablet replica
I20260927 12:44:50.932451 19530 tablet_replica.cc:330] T 0b01206bfc8c4e2eaaaaa512ebf4785b P 4be7338de151428b936ee6fe2aa78195: stopping tablet replica
I20260927 12:44:50.932565 19529 raft_consensus.cc:2229] T 0b01206bfc8c4e2eaaaaa512ebf4785b P 08f3a2e10361467eb16c8716f7710f78 [term 1 LEADER]: Raft consensus shutting down.
I20260927 12:44:50.932891 19530 raft_consensus.cc:2229] T 0b01206bfc8c4e2eaaaaa512ebf4785b P 4be7338de151428b936ee6fe2aa78195 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:44:50.932967 19529 raft_consensus.cc:2258] T 0b01206bfc8c4e2eaaaaa512ebf4785b P 08f3a2e10361467eb16c8716f7710f78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:44:50.933136 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 19098
I20260927 12:44:50.933358 19530 raft_consensus.cc:2258] T 0b01206bfc8c4e2eaaaaa512ebf4785b P 4be7338de151428b936ee6fe2aa78195 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:44:50.935086 19530 ts_tablet_manager.cc:1905] T 0b01206bfc8c4e2eaaaaa512ebf4785b P 4be7338de151428b936ee6fe2aa78195: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:44:50.935385 19528 tablet_replica.cc:330] T 0b01206bfc8c4e2eaaaaa512ebf4785b P 4426197c8af44982baf64877611a2781: stopping tablet replica
I20260927 12:44:50.935653 19528 raft_consensus.cc:2229] T 0b01206bfc8c4e2eaaaaa512ebf4785b P 4426197c8af44982baf64877611a2781 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:44:50.935930 19528 raft_consensus.cc:2258] T 0b01206bfc8c4e2eaaaaa512ebf4785b P 4426197c8af44982baf64877611a2781 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:44:50.937160 19528 ts_tablet_manager.cc:1905] T 0b01206bfc8c4e2eaaaaa512ebf4785b P 4426197c8af44982baf64877611a2781: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:44:50.938732 19530 ts_tablet_manager.cc:1918] T 0b01206bfc8c4e2eaaaaa512ebf4785b P 4be7338de151428b936ee6fe2aa78195: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:44:50.938838 19530 log.cc:1196] T 0b01206bfc8c4e2eaaaaa512ebf4785b P 4be7338de151428b936ee6fe2aa78195: Deleting WAL directory at /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512840836255-392-0/minicluster-data/ts-2/wal/wals/0b01206bfc8c4e2eaaaaa512ebf4785b
I20260927 12:44:50.939155 19530 ts_tablet_manager.cc:1939] T 0b01206bfc8c4e2eaaaaa512ebf4785b P 4be7338de151428b936ee6fe2aa78195: Deleting consensus metadata
I20260927 12:44:50.939466 19528 ts_tablet_manager.cc:1918] T 0b01206bfc8c4e2eaaaaa512ebf4785b P 4426197c8af44982baf64877611a2781: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:44:50.939571 19528 log.cc:1196] T 0b01206bfc8c4e2eaaaaa512ebf4785b P 4426197c8af44982baf64877611a2781: Deleting WAL directory at /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790512840836255-392-0/minicluster-data/ts-1/wal/wals/0b01206bfc8c4e2eaaaaa512ebf4785b
I20260927 12:44:50.939857 19528 ts_tablet_manager.cc:1939] T 0b01206bfc8c4e2eaaaaa512ebf4785b P 4426197c8af44982baf64877611a2781: Deleting consensus metadata
I20260927 12:44:50.940485 19023 catalog_manager.cc:4841] TS 4be7338de151428b936ee6fe2aa78195 (127.0.98.3:39089): tablet 0b01206bfc8c4e2eaaaaa512ebf4785b (table test-table [id=fb9ef00db27d4a7a963a47881556bac7]) successfully deleted
I20260927 12:44:50.940990 19024 catalog_manager.cc:4841] TS 4426197c8af44982baf64877611a2781 (127.0.98.2:41249): tablet 0b01206bfc8c4e2eaaaaa512ebf4785b (table test-table [id=fb9ef00db27d4a7a963a47881556bac7]) successfully deleted
W20260927 12:44:50.941077 19025 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42157 (error 108)
I20260927 12:44:50.941314 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 19229
I20260927 12:44:50.942644 19502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:50.942316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42157 (local address 127.0.0.1:55776)
0927 12:44:50.942455 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:50.942537 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 12:44:50.945767 19025 catalog_manager.cc:4584] TS 08f3a2e10361467eb16c8716f7710f78 (127.0.98.1:42157): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0b01206bfc8c4e2eaaaaa512ebf4785b: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42157: connect: Connection refused (error 111)
I20260927 12:44:50.947089 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 19360
I20260927 12:44:50.952868 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 19007
2026-09-27T12:44:50Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1021 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:44:51 dist-test-slave-bnc1 krb5kdc[19534](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:44:51 dist-test-slave-bnc1 krb5kdc[19534](info): set up 2 sockets
Sep 27 12:44:51 dist-test-slave-bnc1 krb5kdc[19534](info): commencing operation
krb5kdc: starting...
W20260927 12:44:52.029994 392 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/krb5kdc/test-user.keytab.
Sep 27 12:44:52 dist-test-slave-bnc1 krb5kdc[19534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:44:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:44:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:44:52.152927 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45279
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45279
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:44:52.263846 19547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:44:52.264175 19547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:44:52.264257 19547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:44:52.268332 19547 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:44:52.268421 19547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:44:52.268460 19547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:44:52.268514 19547 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:44:52.268550 19547 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:44:52.273963 19547 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:44:52.275058 19547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:44:52.276196 19547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 12:44:52.284979 19547 server_base.cc:1034] running on GCE node
W20260927 12:44:52.284910 19552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:44:52.284958 19555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:44:52.284929 19553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:44:52.285827 19547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:44:52.286923 19547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:44:52.288092 19547 hybrid_clock.cc:648] HybridClock initialized: now 1790513092288062 us; error 43 us; skew 500 ppm
Sep 27 12:44:52 dist-test-slave-bnc1 krb5kdc[19534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:44:52.290949 19547 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:44:52.291771 19547 webserver.cc:460] Webserver started at http://127.0.98.62:38655/ using document root <none> and password file <none>
I20260927 12:44:52.292287 19547 fs_manager.cc:359] Metadata directory not provided
I20260927 12:44:52.292344 19547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:44:52.292505 19547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:44:52.294575 19547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/minicluster-data/master-0/data/instance:
uuid: "ee85d7ed74844acb90916df28aeef3ac"
format_stamp: "Formatted at 2026-09-27 12:44:52 on dist-test-slave-bnc1"
server_key: "5c24819be775ed78cc2d5f7e45da53d2"
server_key_iv: "547d7183b9fc60d064e5afad7c6c5ef2"
server_key_version: "encryptionkey@0"
I20260927 12:44:52.295089 19547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/minicluster-data/master-0/wal/instance:
uuid: "ee85d7ed74844acb90916df28aeef3ac"
format_stamp: "Formatted at 2026-09-27 12:44:52 on dist-test-slave-bnc1"
server_key: "5c24819be775ed78cc2d5f7e45da53d2"
server_key_iv: "547d7183b9fc60d064e5afad7c6c5ef2"
server_key_version: "encryptionkey@0"
I20260927 12:44:52.298882 19547 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260927 12:44:52.301293 19562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:44:52.302572 19547 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:44:52.302706 19547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/minicluster-data/master-0/wal
uuid: "ee85d7ed74844acb90916df28aeef3ac"
format_stamp: "Formatted at 2026-09-27 12:44:52 on dist-test-slave-bnc1"
server_key: "5c24819be775ed78cc2d5f7e45da53d2"
server_key_iv: "547d7183b9fc60d064e5afad7c6c5ef2"
server_key_version: "encryptionkey@0"
I20260927 12:44:52.302816 19547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:44:52.327080 19547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:44:52.330415 19547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:44:52.330623 19547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:44:52.337664 19547 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45279
I20260927 12:44:52.337724 19614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45279 every 8 connection(s)
I20260927 12:44:52.338670 19547 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:44:52.339231 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 19547
I20260927 12:44:52.339375 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/minicluster-data/master-0/wal/instance
I20260927 12:44:52.339661 392 external_mini_cluster.cc:1428] Setting key 760eabb1cd5fc752e60775546ff079f8
I20260927 12:44:52.342132 19615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:44:52.348778 19615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee85d7ed74844acb90916df28aeef3ac: Bootstrap starting.
Sep 27 12:44:52 dist-test-slave-bnc1 krb5kdc[19534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:44:52.351591 19615 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee85d7ed74844acb90916df28aeef3ac: Neither blocks nor log segments found. Creating new log.
I20260927 12:44:52.352497 19615 log.cc:824] T 00000000000000000000000000000000 P ee85d7ed74844acb90916df28aeef3ac: Log is configured to *not* fsync() on all Append() calls
I20260927 12:44:52.354836 19615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee85d7ed74844acb90916df28aeef3ac: No bootstrap required, opened a new log
I20260927 12:44:52.354883 19618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:52.341043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41534 (local address 127.0.98.62:45279)
0927 12:44:52.341462 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:52.341480 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:52.341523 (+ 43us) server_negotiation.cc:407] Connection header received
0927 12:44:52.342405 (+ 882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:52.342428 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:52.342829 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:52.343193 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:44:52.343870 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:52.345380 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:52.346320 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:52.346992 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:52.347171 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:52.349738 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:44:52.349744 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:44:52.349759 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:44:52.349790 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:44:52.351759 (+ 1969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:44:52.352173 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:44:52.352178 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:44:52.352184 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:44:52.352330 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:44:52.352695 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:44:52.352699 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:44:52.352700 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:44:52.353160 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:44:52.353341 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:52.353673 (+ 332us) server_negotiation.cc:299] Negotiation successful
0927 12:44:52.353958 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":118,"threads_started":1}
I20260927 12:44:52.358577 19615 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee85d7ed74844acb90916df28aeef3ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee85d7ed74844acb90916df28aeef3ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45279 } }
I20260927 12:44:52.358801 19615 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee85d7ed74844acb90916df28aeef3ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:44:52.358845 19615 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee85d7ed74844acb90916df28aeef3ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee85d7ed74844acb90916df28aeef3ac, State: Initialized, Role: FOLLOWER
I20260927 12:44:52.359248 19615 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee85d7ed74844acb90916df28aeef3ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee85d7ed74844acb90916df28aeef3ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45279 } }
I20260927 12:44:52.359368 19615 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee85d7ed74844acb90916df28aeef3ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:44:52.359408 19615 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee85d7ed74844acb90916df28aeef3ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:44:52.359478 19615 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee85d7ed74844acb90916df28aeef3ac [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:44:52.360327 19615 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee85d7ed74844acb90916df28aeef3ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee85d7ed74844acb90916df28aeef3ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45279 } }
I20260927 12:44:52.360622 19615 leader_election.cc:302] T 00000000000000000000000000000000 P ee85d7ed74844acb90916df28aeef3ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee85d7ed74844acb90916df28aeef3ac; no voters:
I20260927 12:44:52.360904 19615 leader_election.cc:288] T 00000000000000000000000000000000 P ee85d7ed74844acb90916df28aeef3ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:44:52.361110 19620 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee85d7ed74844acb90916df28aeef3ac [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:44:52.361368 19620 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee85d7ed74844acb90916df28aeef3ac [term 1 LEADER]: Becoming Leader. State: Replica: ee85d7ed74844acb90916df28aeef3ac, State: Running, Role: LEADER
I20260927 12:44:52.361771 19620 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee85d7ed74844acb90916df28aeef3ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee85d7ed74844acb90916df28aeef3ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45279 } }
I20260927 12:44:52.362344 19615 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee85d7ed74844acb90916df28aeef3ac [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:44:52.363579 19621 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee85d7ed74844acb90916df28aeef3ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee85d7ed74844acb90916df28aeef3ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee85d7ed74844acb90916df28aeef3ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45279 } } }
I20260927 12:44:52.363709 19621 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee85d7ed74844acb90916df28aeef3ac [sys.catalog]: This master's current role is: LEADER
I20260927 12:44:52.364099 19629 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:44:52.364043 19622 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee85d7ed74844acb90916df28aeef3ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee85d7ed74844acb90916df28aeef3ac. Latest consensus state: current_term: 1 leader_uuid: "ee85d7ed74844acb90916df28aeef3ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee85d7ed74844acb90916df28aeef3ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45279 } } }
I20260927 12:44:52.364743 19622 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee85d7ed74844acb90916df28aeef3ac [sys.catalog]: This master's current role is: LEADER
I20260927 12:44:52.367447 19629 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:44:52.373286 19629 catalog_manager.cc:1347] Generated new cluster ID: b22f0a2c97ed43e4ad8cef4c19e35bbd
I20260927 12:44:52.373364 19629 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:44:52.379455 19629 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:44:52.380741 19629 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:44:52.391570 19629 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee85d7ed74844acb90916df28aeef3ac: Generated new TSK 0
I20260927 12:44:52.392441 19629 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260927 12:44:52.428234 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45279
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:44:52.540752 19641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:44:52.541046 19641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:44:52.541090 19641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:44:52.545298 19641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:44:52.545367 19641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:44:52.545428 19641 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:44:52.550841 19641 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19641
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:44:52.551937 19641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:44:52.552994 19641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:44:52.562973 19647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:44:52.562971 19649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:44:52.562999 19646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:44:52.563174 19641 server_base.cc:1034] running on GCE node
I20260927 12:44:52.563674 19641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:44:52.564410 19641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:44:52.565660 19641 hybrid_clock.cc:648] HybridClock initialized: now 1790513092565640 us; error 30 us; skew 500 ppm
Sep 27 12:44:52 dist-test-slave-bnc1 krb5kdc[19534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:44:52.568569 19641 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260927 12:44:52.569432 19641 webserver.cc:460] Webserver started at http://127.0.98.1:35391/ using document root <none> and password file <none>
I20260927 12:44:52.570042 19641 fs_manager.cc:359] Metadata directory not provided
I20260927 12:44:52.570137 19641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:44:52.570405 19641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:44:52.572407 19641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/minicluster-data/ts-0/data/instance:
uuid: "fa642e108a0e4067831fb4c08342d549"
format_stamp: "Formatted at 2026-09-27 12:44:52 on dist-test-slave-bnc1"
server_key: "e752f30d8b165a5899a803477c535438"
server_key_iv: "0ba4078f56f53403867c4450b0ca4dc0"
server_key_version: "encryptionkey@0"
I20260927 12:44:52.572962 19641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/minicluster-data/ts-0/wal/instance:
uuid: "fa642e108a0e4067831fb4c08342d549"
format_stamp: "Formatted at 2026-09-27 12:44:52 on dist-test-slave-bnc1"
server_key: "e752f30d8b165a5899a803477c535438"
server_key_iv: "0ba4078f56f53403867c4450b0ca4dc0"
server_key_version: "encryptionkey@0"
I20260927 12:44:52.576627 19641 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 12:44:52.579125 19656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:44:52.580351 19641 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 12:44:52.580483 19641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/minicluster-data/ts-0/wal
uuid: "fa642e108a0e4067831fb4c08342d549"
format_stamp: "Formatted at 2026-09-27 12:44:52 on dist-test-slave-bnc1"
server_key: "e752f30d8b165a5899a803477c535438"
server_key_iv: "0ba4078f56f53403867c4450b0ca4dc0"
server_key_version: "encryptionkey@0"
I20260927 12:44:52.580590 19641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:44:52.600447 19641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:44:52.604224 19641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:44:52.604436 19641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:44:52.605062 19641 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:44:52.606213 19641 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:44:52.606289 19641 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:44:52.606360 19641 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:44:52.606395 19641 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:44:52.617069 19641 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:32987
I20260927 12:44:52.617097 19769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:32987 every 8 connection(s)
I20260927 12:44:52.618050 19641 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:44:52.625031 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 19641
I20260927 12:44:52.625180 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/minicluster-data/ts-0/wal/instance
I20260927 12:44:52.625492 392 external_mini_cluster.cc:1428] Setting key cd78d927a13c7072b382296d56797e12
Sep 27 12:44:52 dist-test-slave-bnc1 krb5kdc[19534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:44:52.632329 19618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:52.619860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42885 (local address 127.0.98.62:45279)
0927 12:44:52.620005 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:52.620009 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:52.620953 (+ 944us) server_negotiation.cc:407] Connection header received
0927 12:44:52.621971 (+ 1018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:52.621974 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:52.622024 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:52.622106 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:44:52.623355 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:52.623902 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:52.625263 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:52.625491 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:52.625640 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:52.628391 (+ 2751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:44:52.628395 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:44:52.628398 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:44:52.628431 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:44:52.629962 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:44:52.630636 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:44:52.630639 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:44:52.630640 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:44:52.630722 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:44:52.631134 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:44:52.631136 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:44:52.631138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:44:52.631314 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:44:52.631392 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:52.632088 (+ 696us) server_negotiation.cc:299] Negotiation successful
0927 12:44:52.632209 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 12:44:52.633842 19772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:52.620279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45279 (local address 127.0.98.1:42885)
0927 12:44:52.620765 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:52.620825 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 12:44:52.621768 (+ 943us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:44:52.622315 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:44:52.622323 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:44:52.622697 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:44:52.623210 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:52.623224 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:52.624008 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:52.624010 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:52.625101 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:52.625151 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:52.625718 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:52.625723 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:52.625924 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:52.626641 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:44:52.626657 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:44:52.628220 (+ 1563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:44:52.630095 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:44:52.630104 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:44:52.630125 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:44:52.630505 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:44:52.630824 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:44:52.630830 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:44:52.630835 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:44:52.631035 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:44:52.631428 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:44:52.631438 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:44:52.631816 (+ 378us) client_negotiation.cc:769] Sending connection context
0927 12:44:52.632060 (+ 244us) client_negotiation.cc:240] Negotiation successful
0927 12:44:52.632467 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":117,"threads_started":1}
I20260927 12:44:52.635245 19770 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45279
I20260927 12:44:52.635506 19770 heartbeater.cc:464] Registering TS with master...
I20260927 12:44:52.636036 19770 heartbeater.cc:511] Master 127.0.98.62:45279 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260927 12:44:52.637542 19579 ts_manager.cc:194] Registered new tserver with Master: fa642e108a0e4067831fb4c08342d549 (127.0.98.1:32987)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260927 12:44:52.638789 19579 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42885
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260927 12:44:52.656271 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45279
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:44:52.779811 19775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:44:52.780102 19775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:44:52.780191 19775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:44:52.784423 19775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:44:52.784504 19775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:44:52.784588 19775 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:44:52.790009 19775 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19775
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:44:52.791163 19775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:44:52.792191 19775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:44:52.801542 19781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:44:52.801749 19775 server_base.cc:1034] running on GCE node
W20260927 12:44:52.801559 19780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:44:52.801548 19783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:44:52.802335 19775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:44:52.802994 19775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:44:52.804180 19775 hybrid_clock.cc:648] HybridClock initialized: now 1790513092804152 us; error 48 us; skew 500 ppm
Sep 27 12:44:52 dist-test-slave-bnc1 krb5kdc[19534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:44:52.807134 19775 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260927 12:44:52.807969 19775 webserver.cc:460] Webserver started at http://127.0.98.2:37819/ using document root <none> and password file <none>
I20260927 12:44:52.808539 19775 fs_manager.cc:359] Metadata directory not provided
I20260927 12:44:52.808627 19775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:44:52.808871 19775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:44:52.810953 19775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/minicluster-data/ts-1/data/instance:
uuid: "1cfa6a1be20745c8a7b93c3070f0ab1f"
format_stamp: "Formatted at 2026-09-27 12:44:52 on dist-test-slave-bnc1"
server_key: "f70e162144000e0aec153fc389202215"
server_key_iv: "3407e7c279c6cce2059e693d36232ef6"
server_key_version: "encryptionkey@0"
I20260927 12:44:52.811512 19775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1cfa6a1be20745c8a7b93c3070f0ab1f"
format_stamp: "Formatted at 2026-09-27 12:44:52 on dist-test-slave-bnc1"
server_key: "f70e162144000e0aec153fc389202215"
server_key_iv: "3407e7c279c6cce2059e693d36232ef6"
server_key_version: "encryptionkey@0"
I20260927 12:44:52.815117 19775 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260927 12:44:52.817423 19790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:44:52.818634 19775 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 12:44:52.818783 19775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/minicluster-data/ts-1/wal
uuid: "1cfa6a1be20745c8a7b93c3070f0ab1f"
format_stamp: "Formatted at 2026-09-27 12:44:52 on dist-test-slave-bnc1"
server_key: "f70e162144000e0aec153fc389202215"
server_key_iv: "3407e7c279c6cce2059e693d36232ef6"
server_key_version: "encryptionkey@0"
I20260927 12:44:52.818893 19775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:44:52.832325 19775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:44:52.835747 19775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:44:52.835948 19775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:44:52.836514 19775 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:44:52.837306 19775 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:44:52.837376 19775 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:44:52.837446 19775 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:44:52.837498 19775 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:44:52.846222 19775 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33453
I20260927 12:44:52.846244 19903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33453 every 8 connection(s)
I20260927 12:44:52.847131 19775 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:44:52.849113 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 19775
I20260927 12:44:52.849252 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/minicluster-data/ts-1/wal/instance
I20260927 12:44:52.849612 392 external_mini_cluster.cc:1428] Setting key dd243c0b6e2a2420c63f15e9a30a083f
Sep 27 12:44:52 dist-test-slave-bnc1 krb5kdc[19534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260927 12:44:52.861966 19618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:52.848973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43135 (local address 127.0.98.62:45279)
0927 12:44:52.849200 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:52.849206 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:52.850174 (+ 968us) server_negotiation.cc:407] Connection header received
0927 12:44:52.851261 (+ 1087us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:52.851264 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:52.851314 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:52.851395 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:44:52.852527 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:52.853043 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:52.854594 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:52.855006 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:52.855146 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:52.858111 (+ 2965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:44:52.858115 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:44:52.858118 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:44:52.858149 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:44:52.859726 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:44:52.860260 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:44:52.860262 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:44:52.860264 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:44:52.860336 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:44:52.860679 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:44:52.860681 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:44:52.860682 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:44:52.860873 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:44:52.860948 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:52.861648 (+ 700us) server_negotiation.cc:299] Negotiation successful
0927 12:44:52.861769 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 12:44:52.862876 19906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:52.849442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45279 (local address 127.0.98.2:43135)
0927 12:44:52.849993 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:52.850061 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0927 12:44:52.851061 (+ 1000us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:44:52.851559 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:44:52.851566 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:44:52.851911 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:44:52.852385 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:52.852399 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:52.853176 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:52.853183 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:52.854439 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:52.854449 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:52.855168 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:52.855172 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:52.855429 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:52.856079 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:44:52.856105 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:44:52.857935 (+ 1830us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:44:52.859848 (+ 1913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:44:52.859853 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:44:52.859876 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:44:52.860126 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:44:52.860437 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:44:52.860439 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:44:52.860440 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:44:52.860576 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:44:52.860967 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:44:52.860972 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:44:52.861355 (+ 383us) client_negotiation.cc:769] Sending connection context
0927 12:44:52.861514 (+ 159us) client_negotiation.cc:240] Negotiation successful
0927 12:44:52.861925 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":190,"threads_started":1}
I20260927 12:44:52.864251 19904 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45279
I20260927 12:44:52.864511 19904 heartbeater.cc:464] Registering TS with master...
I20260927 12:44:52.865139 19904 heartbeater.cc:511] Master 127.0.98.62:45279 requested a full tablet report, sending...
I20260927 12:44:52.866344 19579 ts_manager.cc:194] Registered new tserver with Master: 1cfa6a1be20745c8a7b93c3070f0ab1f (127.0.98.2:33453)
I20260927 12:44:52.866946 19579 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43135
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260927 12:44:52.880905 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45279
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:44:52.994122 19909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:44:52.994408 19909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:44:52.994453 19909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:44:52.998647 19909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:44:52.998708 19909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:44:52.998799 19909 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:44:53.004290 19909 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19909
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:44:53.005389 19909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:44:53.006574 19909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:44:53.017030 19917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:44:53.017015 19914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:44:53.017098 19915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:44:53.017488 19909 server_base.cc:1034] running on GCE node
I20260927 12:44:53.017967 19909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:44:53.018579 19909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:44:53.019749 19909 hybrid_clock.cc:648] HybridClock initialized: now 1790513093019725 us; error 37 us; skew 500 ppm
Sep 27 12:44:53 dist-test-slave-bnc1 krb5kdc[19534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513093, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:44:53.022707 19909 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260927 12:44:53.023674 19909 webserver.cc:460] Webserver started at http://127.0.98.3:44557/ using document root <none> and password file <none>
I20260927 12:44:53.024248 19909 fs_manager.cc:359] Metadata directory not provided
I20260927 12:44:53.024339 19909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:44:53.024569 19909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:44:53.026675 19909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/minicluster-data/ts-2/data/instance:
uuid: "935159cdc4a6485ca3afab17d3f9141a"
format_stamp: "Formatted at 2026-09-27 12:44:53 on dist-test-slave-bnc1"
server_key: "3eb87f45208955f7611bd8f70a7b2a47"
server_key_iv: "3fcacba89e4b96f5bb43e03f6072dca1"
server_key_version: "encryptionkey@0"
I20260927 12:44:53.027247 19909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/minicluster-data/ts-2/wal/instance:
uuid: "935159cdc4a6485ca3afab17d3f9141a"
format_stamp: "Formatted at 2026-09-27 12:44:53 on dist-test-slave-bnc1"
server_key: "3eb87f45208955f7611bd8f70a7b2a47"
server_key_iv: "3fcacba89e4b96f5bb43e03f6072dca1"
server_key_version: "encryptionkey@0"
I20260927 12:44:53.030918 19909 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 12:44:53.033403 19924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:44:53.034689 19909 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:44:53.034825 19909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/minicluster-data/ts-2/wal
uuid: "935159cdc4a6485ca3afab17d3f9141a"
format_stamp: "Formatted at 2026-09-27 12:44:53 on dist-test-slave-bnc1"
server_key: "3eb87f45208955f7611bd8f70a7b2a47"
server_key_iv: "3fcacba89e4b96f5bb43e03f6072dca1"
server_key_version: "encryptionkey@0"
I20260927 12:44:53.034936 19909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:44:53.052902 19909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:44:53.055752 19909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:44:53.055964 19909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:44:53.056561 19909 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:44:53.057451 19909 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:44:53.057523 19909 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:44:53.057637 19909 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:44:53.057693 19909 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:44:53.067337 19909 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33575
I20260927 12:44:53.067379 20037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33575 every 8 connection(s)
I20260927 12:44:53.068252 19909 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
Sep 27 12:44:53 dist-test-slave-bnc1 krb5kdc[19534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513093, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:44:53.077812 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 19909
I20260927 12:44:53.077943 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/minicluster-data/ts-2/wal/instance
I20260927 12:44:53.078255 392 external_mini_cluster.cc:1428] Setting key 1492556f0aa37fdd4b31f2dd2051006d
I20260927 12:44:53.082011 19618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:53.070101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47453 (local address 127.0.98.62:45279)
0927 12:44:53.070246 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:53.070250 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:53.071349 (+ 1099us) server_negotiation.cc:407] Connection header received
0927 12:44:53.072261 (+ 912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:53.072264 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:53.072324 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:53.072426 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:44:53.073631 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:53.074180 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:53.075367 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:53.075590 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:53.075700 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:53.078196 (+ 2496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:44:53.078200 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:44:53.078202 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:44:53.078246 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:44:53.080034 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:44:53.080614 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:44:53.080618 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:44:53.080620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:44:53.080718 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:44:53.081044 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:44:53.081046 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:44:53.081048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:44:53.081216 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:44:53.081291 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:53.081805 (+ 514us) server_negotiation.cc:299] Negotiation successful
0927 12:44:53.081899 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 12:44:53.082855 20040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:53.070513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45279 (local address 127.0.98.3:47453)
0927 12:44:53.071192 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:53.071233 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 12:44:53.072045 (+ 812us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:44:53.072595 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:44:53.072602 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:44:53.072989 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:44:53.073467 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:53.073479 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:53.074307 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:53.074317 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:53.075236 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:53.075245 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:53.075742 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:53.075747 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:53.075912 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:53.076508 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:44:53.076524 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:44:53.078020 (+ 1496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:44:53.080154 (+ 2134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:44:53.080159 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:44:53.080172 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:44:53.080509 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:44:53.080821 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:44:53.080823 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:44:53.080825 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:44:53.080962 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:44:53.081302 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:44:53.081307 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:44:53.081532 (+ 225us) client_negotiation.cc:769] Sending connection context
0927 12:44:53.081759 (+ 227us) client_negotiation.cc:240] Negotiation successful
0927 12:44:53.082008 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":113,"threads_started":1}
I20260927 12:44:53.083978 20038 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45279
I20260927 12:44:53.084216 20038 heartbeater.cc:464] Registering TS with master...
I20260927 12:44:53.084750 20038 heartbeater.cc:511] Master 127.0.98.62:45279 requested a full tablet report, sending...
I20260927 12:44:53.085752 19579 ts_manager.cc:194] Registered new tserver with Master: 935159cdc4a6485ca3afab17d3f9141a (127.0.98.3:33575)
I20260927 12:44:53.086298 19579 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47453
I20260927 12:44:53.092319 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 12:44:53.101023 19618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:53.094400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41536 (local address 127.0.98.62:45279)
0927 12:44:53.094526 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:53.094529 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:53.094728 (+ 199us) server_negotiation.cc:407] Connection header received
0927 12:44:53.094823 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:53.094825 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:53.094859 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:53.094921 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:44:53.095551 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:53.096037 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:53.096826 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:53.097021 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:53.097093 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:53.098439 (+ 1346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:44:53.098442 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:44:53.098444 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:44:53.098463 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:44:53.099672 (+ 1209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:44:53.100035 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:44:53.100038 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:44:53.100039 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:44:53.100108 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:44:53.100335 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:44:53.100337 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:44:53.100338 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:44:53.100530 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:44:53.100628 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:53.100830 (+ 202us) server_negotiation.cc:299] Negotiation successful
0927 12:44:53.100915 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 12:44:53.104115 19579 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41536:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 12:44:53.107205 19579 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:44:53.121273 20049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:53.115395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33453 (local address 127.0.0.1:42992)
0927 12:44:53.116415 (+ 1020us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:53.116429 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:44:53.116510 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:44:53.117355 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:44:53.117358 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:44:53.117374 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:44:53.117483 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:53.117488 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:53.118488 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:53.118493 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:53.119908 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:53.119917 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:53.120826 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:53.120829 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:53.120907 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:53.120938 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 12:44:53.121029 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 12:44:53.121117 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":961,"mutex_wait_us":277,"thread_start_us":106,"threads_started":1}
I20260927 12:44:53.121320 20051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:53.115714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42992 (local address 127.0.98.2:33453)
0927 12:44:53.116880 (+ 1166us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:53.116890 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:53.116908 (+ 18us) server_negotiation.cc:407] Connection header received
0927 12:44:53.116980 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:53.116984 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:53.117119 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:53.117254 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:44:53.117636 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:53.118357 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:53.120031 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:53.120695 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:53.120824 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:53.120972 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:53.121074 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 12:44:53.121168 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1073,"thread_start_us":125,"threads_started":1}
I20260927 12:44:53.122952 20050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:53.115367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33575 (local address 127.0.0.1:44114)
0927 12:44:53.116267 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:53.116326 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 12:44:53.116452 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:44:53.117423 (+ 971us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:44:53.117427 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:44:53.117447 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:44:53.117540 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:53.117545 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:53.118895 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:53.118898 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:53.120394 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:53.120406 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:53.122550 (+ 2144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:53.122554 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:53.122655 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:53.122680 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 12:44:53.122756 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 12:44:53.122825 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":704,"mutex_wait_us":208,"thread_start_us":73,"threads_started":1}
I20260927 12:44:53.123409 20052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:53.115784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32987 (local address 127.0.0.1:60966)
0927 12:44:53.116603 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:53.116617 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:44:53.116711 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:44:53.117176 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:44:53.117182 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:44:53.117203 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:44:53.117349 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:53.117356 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:53.118895 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:53.118898 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:53.120401 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:53.120406 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:53.122982 (+ 2576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:53.122985 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:53.123111 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:53.123133 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 12:44:53.123211 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 12:44:53.123282 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":751,"mutex_wait_us":127,"thread_start_us":86,"threads_started":1}
I20260927 12:44:53.123515 20053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:53.116057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44114 (local address 127.0.98.3:33575)
0927 12:44:53.116773 (+ 716us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:53.116786 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:53.116808 (+ 22us) server_negotiation.cc:407] Connection header received
0927 12:44:53.116888 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:53.116893 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:53.117093 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:53.117247 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:44:53.117693 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:53.118768 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:53.121607 (+ 2839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:53.122431 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:53.122543 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:53.122693 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:53.123282 (+ 589us) server_negotiation.cc:299] Negotiation successful
0927 12:44:53.123378 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":113,"threads_started":1}
I20260927 12:44:53.123899 20054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:53.116195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60966 (local address 127.0.98.1:32987)
0927 12:44:53.116739 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:53.116756 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:53.116781 (+ 25us) server_negotiation.cc:407] Connection header received
0927 12:44:53.116856 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:53.116860 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:53.117017 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:53.117184 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:44:53.117454 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:53.118706 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:53.120533 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:53.121373 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:53.122793 (+ 1420us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:53.122935 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:53.123674 (+ 739us) server_negotiation.cc:299] Negotiation successful
0927 12:44:53.123784 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":94,"threads_started":1}
I20260927 12:44:53.124701 19838 tablet_service.cc:1467] Processing CreateTablet for tablet a3797ecaafd645f0bcba3a6844791373 (DEFAULT_TABLE table=test-table [id=ef30ede0dc4d4d68a7de8d893bb1dd38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:44:53.125918 19838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3797ecaafd645f0bcba3a6844791373. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:44:53.126739 19704 tablet_service.cc:1467] Processing CreateTablet for tablet a3797ecaafd645f0bcba3a6844791373 (DEFAULT_TABLE table=test-table [id=ef30ede0dc4d4d68a7de8d893bb1dd38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:44:53.127043 19971 tablet_service.cc:1467] Processing CreateTablet for tablet a3797ecaafd645f0bcba3a6844791373 (DEFAULT_TABLE table=test-table [id=ef30ede0dc4d4d68a7de8d893bb1dd38]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:44:53.127801 19704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3797ecaafd645f0bcba3a6844791373. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:44:53.127979 19971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3797ecaafd645f0bcba3a6844791373. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:44:53.133703 20055 tablet_bootstrap.cc:492] T a3797ecaafd645f0bcba3a6844791373 P 1cfa6a1be20745c8a7b93c3070f0ab1f: Bootstrap starting.
I20260927 12:44:53.135124 20057 tablet_bootstrap.cc:492] T a3797ecaafd645f0bcba3a6844791373 P fa642e108a0e4067831fb4c08342d549: Bootstrap starting.
I20260927 12:44:53.136992 20055 tablet_bootstrap.cc:654] T a3797ecaafd645f0bcba3a6844791373 P 1cfa6a1be20745c8a7b93c3070f0ab1f: Neither blocks nor log segments found. Creating new log.
I20260927 12:44:53.137297 20057 tablet_bootstrap.cc:654] T a3797ecaafd645f0bcba3a6844791373 P fa642e108a0e4067831fb4c08342d549: Neither blocks nor log segments found. Creating new log.
I20260927 12:44:53.138150 20055 log.cc:824] T a3797ecaafd645f0bcba3a6844791373 P 1cfa6a1be20745c8a7b93c3070f0ab1f: Log is configured to *not* fsync() on all Append() calls
I20260927 12:44:53.138149 20057 log.cc:824] T a3797ecaafd645f0bcba3a6844791373 P fa642e108a0e4067831fb4c08342d549: Log is configured to *not* fsync() on all Append() calls
I20260927 12:44:53.138911 20056 tablet_bootstrap.cc:492] T a3797ecaafd645f0bcba3a6844791373 P 935159cdc4a6485ca3afab17d3f9141a: Bootstrap starting.
I20260927 12:44:53.140307 20055 tablet_bootstrap.cc:492] T a3797ecaafd645f0bcba3a6844791373 P 1cfa6a1be20745c8a7b93c3070f0ab1f: No bootstrap required, opened a new log
I20260927 12:44:53.140307 20057 tablet_bootstrap.cc:492] T a3797ecaafd645f0bcba3a6844791373 P fa642e108a0e4067831fb4c08342d549: No bootstrap required, opened a new log
I20260927 12:44:53.140502 20057 ts_tablet_manager.cc:1397] T a3797ecaafd645f0bcba3a6844791373 P fa642e108a0e4067831fb4c08342d549: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260927 12:44:53.140502 20055 ts_tablet_manager.cc:1397] T a3797ecaafd645f0bcba3a6844791373 P 1cfa6a1be20745c8a7b93c3070f0ab1f: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260927 12:44:53.140818 20056 tablet_bootstrap.cc:654] T a3797ecaafd645f0bcba3a6844791373 P 935159cdc4a6485ca3afab17d3f9141a: Neither blocks nor log segments found. Creating new log.
I20260927 12:44:53.141904 20056 log.cc:824] T a3797ecaafd645f0bcba3a6844791373 P 935159cdc4a6485ca3afab17d3f9141a: Log is configured to *not* fsync() on all Append() calls
I20260927 12:44:53.143638 20056 tablet_bootstrap.cc:492] T a3797ecaafd645f0bcba3a6844791373 P 935159cdc4a6485ca3afab17d3f9141a: No bootstrap required, opened a new log
I20260927 12:44:53.143846 20056 ts_tablet_manager.cc:1397] T a3797ecaafd645f0bcba3a6844791373 P 935159cdc4a6485ca3afab17d3f9141a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260927 12:44:53.144598 20055 raft_consensus.cc:348] T a3797ecaafd645f0bcba3a6844791373 P 1cfa6a1be20745c8a7b93c3070f0ab1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "935159cdc4a6485ca3afab17d3f9141a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33575 } } peers { permanent_uuid: "1cfa6a1be20745c8a7b93c3070f0ab1f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33453 } } peers { permanent_uuid: "fa642e108a0e4067831fb4c08342d549" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32987 } }
I20260927 12:44:53.144896 20055 raft_consensus.cc:374] T a3797ecaafd645f0bcba3a6844791373 P 1cfa6a1be20745c8a7b93c3070f0ab1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:44:53.144960 20055 raft_consensus.cc:729] T a3797ecaafd645f0bcba3a6844791373 P 1cfa6a1be20745c8a7b93c3070f0ab1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cfa6a1be20745c8a7b93c3070f0ab1f, State: Initialized, Role: FOLLOWER
I20260927 12:44:53.144876 20057 raft_consensus.cc:348] T a3797ecaafd645f0bcba3a6844791373 P fa642e108a0e4067831fb4c08342d549 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "935159cdc4a6485ca3afab17d3f9141a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33575 } } peers { permanent_uuid: "1cfa6a1be20745c8a7b93c3070f0ab1f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33453 } } peers { permanent_uuid: "fa642e108a0e4067831fb4c08342d549" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32987 } }
I20260927 12:44:53.145123 20057 raft_consensus.cc:374] T a3797ecaafd645f0bcba3a6844791373 P fa642e108a0e4067831fb4c08342d549 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:44:53.145232 20057 raft_consensus.cc:729] T a3797ecaafd645f0bcba3a6844791373 P fa642e108a0e4067831fb4c08342d549 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa642e108a0e4067831fb4c08342d549, State: Initialized, Role: FOLLOWER
I20260927 12:44:53.145440 20055 consensus_queue.cc:260] T a3797ecaafd645f0bcba3a6844791373 P 1cfa6a1be20745c8a7b93c3070f0ab1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "935159cdc4a6485ca3afab17d3f9141a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33575 } } peers { permanent_uuid: "1cfa6a1be20745c8a7b93c3070f0ab1f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33453 } } peers { permanent_uuid: "fa642e108a0e4067831fb4c08342d549" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32987 } }
I20260927 12:44:53.145668 20057 consensus_queue.cc:260] T a3797ecaafd645f0bcba3a6844791373 P fa642e108a0e4067831fb4c08342d549 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "935159cdc4a6485ca3afab17d3f9141a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33575 } } peers { permanent_uuid: "1cfa6a1be20745c8a7b93c3070f0ab1f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33453 } } peers { permanent_uuid: "fa642e108a0e4067831fb4c08342d549" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32987 } }
I20260927 12:44:53.146219 19904 heartbeater.cc:503] Master 127.0.98.62:45279 was elected leader, sending a full tablet report...
I20260927 12:44:53.146536 20057 ts_tablet_manager.cc:1428] T a3797ecaafd645f0bcba3a6844791373 P fa642e108a0e4067831fb4c08342d549: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 12:44:53.146536 20055 ts_tablet_manager.cc:1428] T a3797ecaafd645f0bcba3a6844791373 P 1cfa6a1be20745c8a7b93c3070f0ab1f: Time spent starting tablet: real 0.006s user 0.004s sys 0.003s
I20260927 12:44:53.146878 19770 heartbeater.cc:503] Master 127.0.98.62:45279 was elected leader, sending a full tablet report...
I20260927 12:44:53.147765 20056 raft_consensus.cc:348] T a3797ecaafd645f0bcba3a6844791373 P 935159cdc4a6485ca3afab17d3f9141a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "935159cdc4a6485ca3afab17d3f9141a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33575 } } peers { permanent_uuid: "1cfa6a1be20745c8a7b93c3070f0ab1f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33453 } } peers { permanent_uuid: "fa642e108a0e4067831fb4c08342d549" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32987 } }
I20260927 12:44:53.148043 20056 raft_consensus.cc:374] T a3797ecaafd645f0bcba3a6844791373 P 935159cdc4a6485ca3afab17d3f9141a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:44:53.148103 20056 raft_consensus.cc:729] T a3797ecaafd645f0bcba3a6844791373 P 935159cdc4a6485ca3afab17d3f9141a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 935159cdc4a6485ca3afab17d3f9141a, State: Initialized, Role: FOLLOWER
I20260927 12:44:53.148537 20056 consensus_queue.cc:260] T a3797ecaafd645f0bcba3a6844791373 P 935159cdc4a6485ca3afab17d3f9141a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "935159cdc4a6485ca3afab17d3f9141a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33575 } } peers { permanent_uuid: "1cfa6a1be20745c8a7b93c3070f0ab1f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33453 } } peers { permanent_uuid: "fa642e108a0e4067831fb4c08342d549" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32987 } }
I20260927 12:44:53.149168 20038 heartbeater.cc:503] Master 127.0.98.62:45279 was elected leader, sending a full tablet report...
I20260927 12:44:53.149509 20056 ts_tablet_manager.cc:1428] T a3797ecaafd645f0bcba3a6844791373 P 935159cdc4a6485ca3afab17d3f9141a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260927 12:44:53.252285 20062 raft_consensus.cc:482] T a3797ecaafd645f0bcba3a6844791373 P fa642e108a0e4067831fb4c08342d549 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:44:53.252488 20062 raft_consensus.cc:504] T a3797ecaafd645f0bcba3a6844791373 P fa642e108a0e4067831fb4c08342d549 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "935159cdc4a6485ca3afab17d3f9141a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33575 } } peers { permanent_uuid: "1cfa6a1be20745c8a7b93c3070f0ab1f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33453 } } peers { permanent_uuid: "fa642e108a0e4067831fb4c08342d549" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32987 } }
I20260927 12:44:53.253623 20062 leader_election.cc:288] T a3797ecaafd645f0bcba3a6844791373 P fa642e108a0e4067831fb4c08342d549 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 935159cdc4a6485ca3afab17d3f9141a (127.0.98.3:33575), 1cfa6a1be20745c8a7b93c3070f0ab1f (127.0.98.2:33453)
I20260927 12:44:53.259166 20051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:53.253957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46227 (local address 127.0.98.2:33453)
0927 12:44:53.254087 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:53.254092 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:53.254397 (+ 305us) server_negotiation.cc:407] Connection header received
0927 12:44:53.254543 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:53.254546 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:53.254598 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:53.254693 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:44:53.255150 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:53.255936 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:53.257816 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:53.258457 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:53.258570 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:53.258634 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:53.258964 (+ 330us) server_negotiation.cc:299] Negotiation successful
0927 12:44:53.259032 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 12:44:53.259174 20053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:53.253976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44107 (local address 127.0.98.3:33575)
0927 12:44:53.254107 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:53.254112 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:53.254466 (+ 354us) server_negotiation.cc:407] Connection header received
0927 12:44:53.254542 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:53.254545 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:53.254598 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:53.254689 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:44:53.255150 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:53.255927 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:53.257816 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:53.258463 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:53.258570 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:53.258634 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:53.258964 (+ 330us) server_negotiation.cc:299] Negotiation successful
0927 12:44:53.259032 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 12:44:53.259265 20064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:53.253872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33453 (local address 127.0.98.1:46227)
0927 12:44:53.254286 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:53.254301 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:44:53.254430 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:44:53.254724 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:44:53.254727 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:44:53.254768 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:44:53.255038 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:53.255043 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:53.256048 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:53.256052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:53.257682 (+ 1630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:53.257694 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:53.258601 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:53.258604 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:53.258801 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:53.258863 (+ 62us) client_negotiation.cc:769] Sending connection context
0927 12:44:53.258932 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 12:44:53.259063 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":337,"spinlock_wait_cycles":8576,"thread_start_us":132,"threads_started":1}
I20260927 12:44:53.259353 20065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:53.253891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33575 (local address 127.0.98.1:44107)
0927 12:44:53.254370 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:53.254382 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 12:44:53.254462 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:44:53.254724 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:44:53.254727 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:44:53.254766 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:44:53.255030 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:53.255039 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:53.256048 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:53.256052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:53.257682 (+ 1630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:53.257694 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:53.258601 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:53.258604 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:53.258774 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:53.258852 (+ 78us) client_negotiation.cc:769] Sending connection context
0927 12:44:53.258932 (+ 80us) client_negotiation.cc:240] Negotiation successful
0927 12:44:53.259063 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"mutex_wait_us":291,"spinlock_wait_cycles":30080,"thread_start_us":114,"threads_started":1}
I20260927 12:44:53.259925 19858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3797ecaafd645f0bcba3a6844791373" candidate_uuid: "fa642e108a0e4067831fb4c08342d549" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cfa6a1be20745c8a7b93c3070f0ab1f" is_pre_election: true
I20260927 12:44:53.259927 19992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3797ecaafd645f0bcba3a6844791373" candidate_uuid: "fa642e108a0e4067831fb4c08342d549" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "935159cdc4a6485ca3afab17d3f9141a" is_pre_election: true
I20260927 12:44:53.260211 19992 raft_consensus.cc:2454] T a3797ecaafd645f0bcba3a6844791373 P 935159cdc4a6485ca3afab17d3f9141a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa642e108a0e4067831fb4c08342d549 in term 0.
I20260927 12:44:53.260234 19858 raft_consensus.cc:2454] T a3797ecaafd645f0bcba3a6844791373 P 1cfa6a1be20745c8a7b93c3070f0ab1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa642e108a0e4067831fb4c08342d549 in term 0.
I20260927 12:44:53.260686 19657 leader_election.cc:302] T a3797ecaafd645f0bcba3a6844791373 P fa642e108a0e4067831fb4c08342d549 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 935159cdc4a6485ca3afab17d3f9141a, fa642e108a0e4067831fb4c08342d549; no voters:
I20260927 12:44:53.260988 20062 raft_consensus.cc:2781] T a3797ecaafd645f0bcba3a6844791373 P fa642e108a0e4067831fb4c08342d549 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:44:53.261067 20062 raft_consensus.cc:482] T a3797ecaafd645f0bcba3a6844791373 P fa642e108a0e4067831fb4c08342d549 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:44:53.261098 20062 raft_consensus.cc:3037] T a3797ecaafd645f0bcba3a6844791373 P fa642e108a0e4067831fb4c08342d549 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:44:53.262184 20062 raft_consensus.cc:504] T a3797ecaafd645f0bcba3a6844791373 P fa642e108a0e4067831fb4c08342d549 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "935159cdc4a6485ca3afab17d3f9141a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33575 } } peers { permanent_uuid: "1cfa6a1be20745c8a7b93c3070f0ab1f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33453 } } peers { permanent_uuid: "fa642e108a0e4067831fb4c08342d549" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32987 } }
I20260927 12:44:53.262560 20062 leader_election.cc:288] T a3797ecaafd645f0bcba3a6844791373 P fa642e108a0e4067831fb4c08342d549 [CANDIDATE]: Term 1 election: Requested vote from peers 935159cdc4a6485ca3afab17d3f9141a (127.0.98.3:33575), 1cfa6a1be20745c8a7b93c3070f0ab1f (127.0.98.2:33453)
I20260927 12:44:53.262924 19858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3797ecaafd645f0bcba3a6844791373" candidate_uuid: "fa642e108a0e4067831fb4c08342d549" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cfa6a1be20745c8a7b93c3070f0ab1f"
I20260927 12:44:53.262970 19992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3797ecaafd645f0bcba3a6844791373" candidate_uuid: "fa642e108a0e4067831fb4c08342d549" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "935159cdc4a6485ca3afab17d3f9141a"
I20260927 12:44:53.263036 19858 raft_consensus.cc:3037] T a3797ecaafd645f0bcba3a6844791373 P 1cfa6a1be20745c8a7b93c3070f0ab1f [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:44:53.263109 19992 raft_consensus.cc:3037] T a3797ecaafd645f0bcba3a6844791373 P 935159cdc4a6485ca3afab17d3f9141a [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:44:53.264245 19992 raft_consensus.cc:2454] T a3797ecaafd645f0bcba3a6844791373 P 935159cdc4a6485ca3afab17d3f9141a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa642e108a0e4067831fb4c08342d549 in term 1.
I20260927 12:44:53.264271 19858 raft_consensus.cc:2454] T a3797ecaafd645f0bcba3a6844791373 P 1cfa6a1be20745c8a7b93c3070f0ab1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa642e108a0e4067831fb4c08342d549 in term 1.
I20260927 12:44:53.264545 19657 leader_election.cc:302] T a3797ecaafd645f0bcba3a6844791373 P fa642e108a0e4067831fb4c08342d549 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 935159cdc4a6485ca3afab17d3f9141a, fa642e108a0e4067831fb4c08342d549; no voters:
I20260927 12:44:53.264742 20062 raft_consensus.cc:2781] T a3797ecaafd645f0bcba3a6844791373 P fa642e108a0e4067831fb4c08342d549 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:44:53.264994 20062 raft_consensus.cc:686] T a3797ecaafd645f0bcba3a6844791373 P fa642e108a0e4067831fb4c08342d549 [term 1 LEADER]: Becoming Leader. State: Replica: fa642e108a0e4067831fb4c08342d549, State: Running, Role: LEADER
I20260927 12:44:53.265350 20062 consensus_queue.cc:237] T a3797ecaafd645f0bcba3a6844791373 P fa642e108a0e4067831fb4c08342d549 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "935159cdc4a6485ca3afab17d3f9141a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33575 } } peers { permanent_uuid: "1cfa6a1be20745c8a7b93c3070f0ab1f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33453 } } peers { permanent_uuid: "fa642e108a0e4067831fb4c08342d549" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32987 } }
I20260927 12:44:53.268635 19578 catalog_manager.cc:5495] T a3797ecaafd645f0bcba3a6844791373 P fa642e108a0e4067831fb4c08342d549 reported cstate change: term changed from 0 to 1, leader changed from <none> to fa642e108a0e4067831fb4c08342d549 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "fa642e108a0e4067831fb4c08342d549" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "935159cdc4a6485ca3afab17d3f9141a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cfa6a1be20745c8a7b93c3070f0ab1f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa642e108a0e4067831fb4c08342d549" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32987 } health_report { overall_health: HEALTHY } } }
I20260927 12:44:53.310840 20054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:53.307139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60968 (local address 127.0.98.1:32987)
0927 12:44:53.307298 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:53.307303 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:53.307319 (+ 16us) server_negotiation.cc:407] Connection header received
0927 12:44:53.307452 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:53.307454 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:53.307508 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:53.307624 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:44:53.307908 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:53.308425 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:53.309295 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:53.309516 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:53.309687 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:53.310007 (+ 320us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:44:53.310433 (+ 426us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:44:53.310574 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:53.310618 (+ 44us) server_negotiation.cc:299] Negotiation successful
0927 12:44:53.310684 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 12:44:53.316704 19858 raft_consensus.cc:1261] T a3797ecaafd645f0bcba3a6844791373 P 1cfa6a1be20745c8a7b93c3070f0ab1f [term 1 FOLLOWER]: Refusing update from remote peer fa642e108a0e4067831fb4c08342d549: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:44:53.316704 19992 raft_consensus.cc:1261] T a3797ecaafd645f0bcba3a6844791373 P 935159cdc4a6485ca3afab17d3f9141a [term 1 FOLLOWER]: Refusing update from remote peer fa642e108a0e4067831fb4c08342d549: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:44:53.317456 20062 consensus_queue.cc:1035] T a3797ecaafd645f0bcba3a6844791373 P fa642e108a0e4067831fb4c08342d549 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cfa6a1be20745c8a7b93c3070f0ab1f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:44:53.317658 20066 consensus_queue.cc:1035] T a3797ecaafd645f0bcba3a6844791373 P fa642e108a0e4067831fb4c08342d549 [LEADER]: Connected to new peer: Peer: permanent_uuid: "935159cdc4a6485ca3afab17d3f9141a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 12:44:53.319214 20039 tablet.cc:2369] T a3797ecaafd645f0bcba3a6844791373 P 935159cdc4a6485ca3afab17d3f9141a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:44:53.333046 19578 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:44:53.333222 19578 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:44:53.335575 19578 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ee85d7ed74844acb90916df28aeef3ac: Sending DeleteTablet for 3 replicas of tablet a3797ecaafd645f0bcba3a6844791373
I20260927 12:44:53.336338 19971 tablet_service.cc:1514] Processing DeleteTablet for tablet a3797ecaafd645f0bcba3a6844791373 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:44:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44114
I20260927 12:44:53.336434 19704 tablet_service.cc:1514] Processing DeleteTablet for tablet a3797ecaafd645f0bcba3a6844791373 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:44:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60966
I20260927 12:44:53.336818 19838 tablet_service.cc:1514] Processing DeleteTablet for tablet a3797ecaafd645f0bcba3a6844791373 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:44:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42992
I20260927 12:44:53.336891 20080 tablet_replica.cc:330] T a3797ecaafd645f0bcba3a6844791373 P fa642e108a0e4067831fb4c08342d549: stopping tablet replica
I20260927 12:44:53.336985 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 19641
I20260927 12:44:53.337307 20080 raft_consensus.cc:2229] T a3797ecaafd645f0bcba3a6844791373 P fa642e108a0e4067831fb4c08342d549 [term 1 LEADER]: Raft consensus shutting down.
I20260927 12:44:53.338474 20081 tablet_replica.cc:330] T a3797ecaafd645f0bcba3a6844791373 P 1cfa6a1be20745c8a7b93c3070f0ab1f: stopping tablet replica
I20260927 12:44:53.338790 20081 raft_consensus.cc:2229] T a3797ecaafd645f0bcba3a6844791373 P 1cfa6a1be20745c8a7b93c3070f0ab1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:44:53.339150 20081 raft_consensus.cc:2258] T a3797ecaafd645f0bcba3a6844791373 P 1cfa6a1be20745c8a7b93c3070f0ab1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:44:53.339515 20079 tablet_replica.cc:330] T a3797ecaafd645f0bcba3a6844791373 P 935159cdc4a6485ca3afab17d3f9141a: stopping tablet replica
I20260927 12:44:53.339766 20079 raft_consensus.cc:2229] T a3797ecaafd645f0bcba3a6844791373 P 935159cdc4a6485ca3afab17d3f9141a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:44:53.340011 20079 raft_consensus.cc:2258] T a3797ecaafd645f0bcba3a6844791373 P 935159cdc4a6485ca3afab17d3f9141a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:44:53.341253 20081 ts_tablet_manager.cc:1905] T a3797ecaafd645f0bcba3a6844791373 P 1cfa6a1be20745c8a7b93c3070f0ab1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:44:53.341274 20079 ts_tablet_manager.cc:1905] T a3797ecaafd645f0bcba3a6844791373 P 935159cdc4a6485ca3afab17d3f9141a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:44:53.343982 20079 ts_tablet_manager.cc:1918] T a3797ecaafd645f0bcba3a6844791373 P 935159cdc4a6485ca3afab17d3f9141a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:44:53.344082 20079 log.cc:1196] T a3797ecaafd645f0bcba3a6844791373 P 935159cdc4a6485ca3afab17d3f9141a: Deleting WAL directory at /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/minicluster-data/ts-2/wal/wals/a3797ecaafd645f0bcba3a6844791373
I20260927 12:44:53.344375 20079 ts_tablet_manager.cc:1939] T a3797ecaafd645f0bcba3a6844791373 P 935159cdc4a6485ca3afab17d3f9141a: Deleting consensus metadata
I20260927 12:44:53.344584 20081 ts_tablet_manager.cc:1918] T a3797ecaafd645f0bcba3a6844791373 P 1cfa6a1be20745c8a7b93c3070f0ab1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:44:53.344693 20081 log.cc:1196] T a3797ecaafd645f0bcba3a6844791373 P 1cfa6a1be20745c8a7b93c3070f0ab1f: Deleting WAL directory at /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790512840836255-392-0/minicluster-data/ts-1/wal/wals/a3797ecaafd645f0bcba3a6844791373
I20260927 12:44:53.345009 20081 ts_tablet_manager.cc:1939] T a3797ecaafd645f0bcba3a6844791373 P 1cfa6a1be20745c8a7b93c3070f0ab1f: Deleting consensus metadata
I20260927 12:44:53.345613 19563 catalog_manager.cc:4841] TS 935159cdc4a6485ca3afab17d3f9141a (127.0.98.3:33575): tablet a3797ecaafd645f0bcba3a6844791373 (table test-table [id=ef30ede0dc4d4d68a7de8d893bb1dd38]) successfully deleted
I20260927 12:44:53.346249 19566 catalog_manager.cc:4841] TS 1cfa6a1be20745c8a7b93c3070f0ab1f (127.0.98.2:33453): tablet a3797ecaafd645f0bcba3a6844791373 (table test-table [id=ef30ede0dc4d4d68a7de8d893bb1dd38]) successfully deleted
W20260927 12:44:53.347568 19564 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:32987 (error 108)
I20260927 12:44:53.348616 20052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:53.348285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32987 (local address 127.0.0.1:60974)
0927 12:44:53.348412 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:53.348513 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:32987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 12:44:53.348696 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 19775
W20260927 12:44:53.348906 19564 catalog_manager.cc:4584] TS fa642e108a0e4067831fb4c08342d549 (127.0.98.1:32987): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a3797ecaafd645f0bcba3a6844791373: Network error: Client connection negotiation failed: client connection to 127.0.98.1:32987: connect: Connection refused (error 111)
I20260927 12:44:53.355288 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 19909
I20260927 12:44:53.360673 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 19547
2026-09-27T12:44:53Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2408 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-27T12:44:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:44:53Z Disabled control of system clock
I20260927 12:44:53.391018 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512840836255-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512840836255-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512840836255-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40049
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40049
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 12:44:53.499809 20086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:44:53.500046 20086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:44:53.500080 20086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:44:53.503939 20086 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:44:53.503993 20086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:44:53.504007 20086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:44:53.504024 20086 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:44:53.504038 20086 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:44:53.508562 20086 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512840836255-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512840836255-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40049
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512840836255-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:44:53.509506 20086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:44:53.510890 20086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:44:53.519299 20091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:44:53.519294 20092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:44:53.519554 20086 server_base.cc:1034] running on GCE node
W20260927 12:44:53.519294 20094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:44:53.520280 20086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:44:53.521340 20086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:44:53.522511 20086 hybrid_clock.cc:648] HybridClock initialized: now 1790513093522481 us; error 44 us; skew 500 ppm
I20260927 12:44:53.524457 20086 webserver.cc:460] Webserver started at http://127.0.98.62:41125/ using document root <none> and password file <none>
I20260927 12:44:53.525074 20086 fs_manager.cc:359] Metadata directory not provided
I20260927 12:44:53.525136 20086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:44:53.525370 20086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:44:53.527470 20086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512840836255-392-0/minicluster-data/master-0/data/instance:
uuid: "f678ce7941b44d3a984688043d96e789"
format_stamp: "Formatted at 2026-09-27 12:44:53 on dist-test-slave-bnc1"
server_key: "9c4014103a949e7c62cac6fa4c875898"
server_key_iv: "08ca468e0b6768120084766d94f104f1"
server_key_version: "encryptionkey@0"
I20260927 12:44:53.528020 20086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512840836255-392-0/minicluster-data/master-0/wal/instance:
uuid: "f678ce7941b44d3a984688043d96e789"
format_stamp: "Formatted at 2026-09-27 12:44:53 on dist-test-slave-bnc1"
server_key: "9c4014103a949e7c62cac6fa4c875898"
server_key_iv: "08ca468e0b6768120084766d94f104f1"
server_key_version: "encryptionkey@0"
I20260927 12:44:53.531792 20086 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 12:44:53.534297 20100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:44:53.535547 20086 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 12:44:53.535712 20086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512840836255-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512840836255-392-0/minicluster-data/master-0/wal
uuid: "f678ce7941b44d3a984688043d96e789"
format_stamp: "Formatted at 2026-09-27 12:44:53 on dist-test-slave-bnc1"
server_key: "9c4014103a949e7c62cac6fa4c875898"
server_key_iv: "08ca468e0b6768120084766d94f104f1"
server_key_version: "encryptionkey@0"
I20260927 12:44:53.535822 20086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512840836255-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512840836255-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512840836255-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:44:53.550387 20086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:44:53.551045 20086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:44:53.551247 20086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:44:53.558005 20086 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40049
I20260927 12:44:53.558017 20152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40049 every 8 connection(s)
I20260927 12:44:53.558921 20086 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:44:53.561904 20153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:44:53.566434 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 20086
I20260927 12:44:53.566613 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512840836255-392-0/minicluster-data/master-0/wal/instance
I20260927 12:44:53.566891 392 external_mini_cluster.cc:1428] Setting key b66a3e3a10beb45648e0ecd066ad72b2
I20260927 12:44:53.567420 20153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f678ce7941b44d3a984688043d96e789: Bootstrap starting.
I20260927 12:44:53.569903 20153 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f678ce7941b44d3a984688043d96e789: Neither blocks nor log segments found. Creating new log.
I20260927 12:44:53.570613 20153 log.cc:824] T 00000000000000000000000000000000 P f678ce7941b44d3a984688043d96e789: Log is configured to *not* fsync() on all Append() calls
I20260927 12:44:53.572358 20153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f678ce7941b44d3a984688043d96e789: No bootstrap required, opened a new log
I20260927 12:44:53.575868 20156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:53.568300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39546 (local address 127.0.98.62:40049)
0927 12:44:53.568949 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:53.568963 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:53.568998 (+ 35us) server_negotiation.cc:407] Connection header received
0927 12:44:53.569705 (+ 707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:53.569722 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:53.570169 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:53.570498 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:44:53.570906 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:53.571846 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:53.572775 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:53.573237 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:53.573419 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:53.574280 (+ 861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:44:53.574286 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:44:53.574297 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:44:53.574661 (+ 364us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:44:53.574712 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:44:53.574731 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:44:53.574838 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:53.574948 (+ 110us) server_negotiation.cc:299] Negotiation successful
0927 12:44:53.575198 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":150,"threads_started":1}
I20260927 12:44:53.575762 20153 raft_consensus.cc:348] T 00000000000000000000000000000000 P f678ce7941b44d3a984688043d96e789 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f678ce7941b44d3a984688043d96e789" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40049 } }
I20260927 12:44:53.575986 20153 raft_consensus.cc:374] T 00000000000000000000000000000000 P f678ce7941b44d3a984688043d96e789 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:44:53.576047 20153 raft_consensus.cc:729] T 00000000000000000000000000000000 P f678ce7941b44d3a984688043d96e789 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f678ce7941b44d3a984688043d96e789, State: Initialized, Role: FOLLOWER
I20260927 12:44:53.576511 20153 consensus_queue.cc:260] T 00000000000000000000000000000000 P f678ce7941b44d3a984688043d96e789 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f678ce7941b44d3a984688043d96e789" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40049 } }
I20260927 12:44:53.576646 20153 raft_consensus.cc:388] T 00000000000000000000000000000000 P f678ce7941b44d3a984688043d96e789 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:44:53.576697 20153 raft_consensus.cc:482] T 00000000000000000000000000000000 P f678ce7941b44d3a984688043d96e789 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:44:53.576777 20153 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f678ce7941b44d3a984688043d96e789 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:44:53.577752 20153 raft_consensus.cc:504] T 00000000000000000000000000000000 P f678ce7941b44d3a984688043d96e789 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f678ce7941b44d3a984688043d96e789" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40049 } }
I20260927 12:44:53.578126 20153 leader_election.cc:302] T 00000000000000000000000000000000 P f678ce7941b44d3a984688043d96e789 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f678ce7941b44d3a984688043d96e789; no voters:
I20260927 12:44:53.578455 20153 leader_election.cc:288] T 00000000000000000000000000000000 P f678ce7941b44d3a984688043d96e789 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:44:53.578562 20158 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f678ce7941b44d3a984688043d96e789 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:44:53.578814 20158 raft_consensus.cc:686] T 00000000000000000000000000000000 P f678ce7941b44d3a984688043d96e789 [term 1 LEADER]: Becoming Leader. State: Replica: f678ce7941b44d3a984688043d96e789, State: Running, Role: LEADER
I20260927 12:44:53.579173 20158 consensus_queue.cc:237] T 00000000000000000000000000000000 P f678ce7941b44d3a984688043d96e789 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f678ce7941b44d3a984688043d96e789" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40049 } }
I20260927 12:44:53.579658 20153 sys_catalog.cc:564] T 00000000000000000000000000000000 P f678ce7941b44d3a984688043d96e789 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:44:53.580842 20159 sys_catalog.cc:455] T 00000000000000000000000000000000 P f678ce7941b44d3a984688043d96e789 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f678ce7941b44d3a984688043d96e789" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f678ce7941b44d3a984688043d96e789" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40049 } } }
I20260927 12:44:53.580960 20159 sys_catalog.cc:458] T 00000000000000000000000000000000 P f678ce7941b44d3a984688043d96e789 [sys.catalog]: This master's current role is: LEADER
I20260927 12:44:53.581108 20160 sys_catalog.cc:455] T 00000000000000000000000000000000 P f678ce7941b44d3a984688043d96e789 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f678ce7941b44d3a984688043d96e789. Latest consensus state: current_term: 1 leader_uuid: "f678ce7941b44d3a984688043d96e789" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f678ce7941b44d3a984688043d96e789" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40049 } } }
I20260927 12:44:53.581192 20160 sys_catalog.cc:458] T 00000000000000000000000000000000 P f678ce7941b44d3a984688043d96e789 [sys.catalog]: This master's current role is: LEADER
I20260927 12:44:53.581408 20167 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:44:53.584461 20167 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:44:53.589999 20167 catalog_manager.cc:1347] Generated new cluster ID: a1a71669fc7741f9b4147ba098a4eb55
I20260927 12:44:53.590065 20167 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:44:53.600780 20167 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:44:53.601738 20167 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:44:53.607714 20167 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f678ce7941b44d3a984688043d96e789: Generated new TSK 0
I20260927 12:44:53.608434 20167 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:44:53.619042 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512840836255-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512840836255-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512840836255-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40049
--builtin_ntp_servers=127.0.98.20:37733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 12:44:53.734156 20177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:44:53.734432 20177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:44:53.734472 20177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:44:53.738737 20177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:44:53.738803 20177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:44:53.738862 20177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260927 12:44:53.743785 20177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512840836255-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512840836255-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40049
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20177
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512840836255-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:44:53.744896 20177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:44:53.746378 20177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:44:53.755928 20183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:44:53.756050 20177 server_base.cc:1034] running on GCE node
W20260927 12:44:53.755959 20182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:44:53.755960 20185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:44:53.756541 20177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:44:53.757175 20177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:44:53.758354 20177 hybrid_clock.cc:648] HybridClock initialized: now 1790513093758335 us; error 26 us; skew 500 ppm
I20260927 12:44:53.760483 20177 webserver.cc:460] Webserver started at http://127.0.98.1:34935/ using document root <none> and password file <none>
I20260927 12:44:53.761047 20177 fs_manager.cc:359] Metadata directory not provided
I20260927 12:44:53.761106 20177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:44:53.761268 20177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:44:53.763296 20177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512840836255-392-0/minicluster-data/ts-0/data/instance:
uuid: "14d7a46b190546d59b14725d3c378176"
format_stamp: "Formatted at 2026-09-27 12:44:53 on dist-test-slave-bnc1"
server_key: "9dddafe61cb116a7bf8c519a94090254"
server_key_iv: "c974da37868bada42384305df5300535"
server_key_version: "encryptionkey@0"
I20260927 12:44:53.763801 20177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512840836255-392-0/minicluster-data/ts-0/wal/instance:
uuid: "14d7a46b190546d59b14725d3c378176"
format_stamp: "Formatted at 2026-09-27 12:44:53 on dist-test-slave-bnc1"
server_key: "9dddafe61cb116a7bf8c519a94090254"
server_key_iv: "c974da37868bada42384305df5300535"
server_key_version: "encryptionkey@0"
I20260927 12:44:53.767321 20177 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260927 12:44:53.769639 20191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:44:53.770852 20177 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 12:44:53.770942 20177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512840836255-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512840836255-392-0/minicluster-data/ts-0/wal
uuid: "14d7a46b190546d59b14725d3c378176"
format_stamp: "Formatted at 2026-09-27 12:44:53 on dist-test-slave-bnc1"
server_key: "9dddafe61cb116a7bf8c519a94090254"
server_key_iv: "c974da37868bada42384305df5300535"
server_key_version: "encryptionkey@0"
I20260927 12:44:53.771049 20177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512840836255-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512840836255-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512840836255-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:44:53.803334 20177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:44:53.804059 20177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:44:53.804214 20177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:44:53.804781 20177 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:44:53.805732 20177 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:44:53.805781 20177 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:44:53.805822 20177 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:44:53.805837 20177 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:44:53.814527 20177 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44097
I20260927 12:44:53.814620 20304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44097 every 8 connection(s)
I20260927 12:44:53.815518 20177 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512840836255-392-0/minicluster-data/ts-0/data/info.pb
I20260927 12:44:53.816157 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 20177
I20260927 12:44:53.816289 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512840836255-392-0/minicluster-data/ts-0/wal/instance
I20260927 12:44:53.816596 392 external_mini_cluster.cc:1428] Setting key b7f785cc369b3c8d95a67bb0be23287e
I20260927 12:44:53.818884 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512840836255-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512840836255-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512840836255-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40049
--builtin_ntp_servers=127.0.98.20:37733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 12:44:53.826436 20156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:53.817638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55235 (local address 127.0.98.62:40049)
0927 12:44:53.817770 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:53.817775 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:53.819147 (+ 1372us) server_negotiation.cc:407] Connection header received
0927 12:44:53.820629 (+ 1482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:53.820635 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:53.820698 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:53.820816 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:44:53.821834 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:53.822763 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:53.823980 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:53.824422 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:53.824560 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:53.825792 (+ 1232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:44:53.825799 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:44:53.825805 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:44:53.825825 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:44:53.825852 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:44:53.825867 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:44:53.825959 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:53.826127 (+ 168us) server_negotiation.cc:299] Negotiation successful
0927 12:44:53.826216 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 12:44:53.827242 20307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:53.818162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40049 (local address 127.0.98.1:55235)
0927 12:44:53.818907 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:53.818983 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0927 12:44:53.820407 (+ 1424us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:44:53.820970 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:44:53.820977 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:44:53.821100 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:44:53.821690 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:53.821706 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:53.822903 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:53.822906 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:53.823842 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:53.823853 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:53.824586 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:53.824589 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:53.824744 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:53.825439 (+ 695us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 12:44:53.825445 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:44:53.825486 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 12:44:53.825552 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:44:53.825958 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:44:53.825966 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:44:53.825970 (+ 4us) client_negotiation.cc:769] Sending connection context
0927 12:44:53.826074 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 12:44:53.826355 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":132,"threads_started":1}
I20260927 12:44:53.828552 20305 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40049
I20260927 12:44:53.828843 20305 heartbeater.cc:464] Registering TS with master...
I20260927 12:44:53.829587 20305 heartbeater.cc:511] Master 127.0.98.62:40049 requested a full tablet report, sending...
I20260927 12:44:53.831275 20117 ts_manager.cc:194] Registered new tserver with Master: 14d7a46b190546d59b14725d3c378176 (127.0.98.1:44097)
I20260927 12:44:53.832470 20117 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:55235
W20260927 12:44:53.933328 20308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:44:53.933661 20308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:44:53.933707 20308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:44:53.937790 20308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:44:53.937848 20308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:44:53.937904 20308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260927 12:44:53.942730 20308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512840836255-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512840836255-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40049
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20308
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512840836255-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:44:53.943864 20308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:44:53.945302 20308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:44:53.954972 20313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:44:53.954955 20314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:44:53.954943 20316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:44:53.955324 20308 server_base.cc:1034] running on GCE node
I20260927 12:44:53.955745 20308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:44:53.956305 20308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:44:53.957472 20308 hybrid_clock.cc:648] HybridClock initialized: now 1790513093957460 us; error 23 us; skew 500 ppm
I20260927 12:44:53.959200 20308 webserver.cc:460] Webserver started at http://127.0.98.2:39701/ using document root <none> and password file <none>
I20260927 12:44:53.959717 20308 fs_manager.cc:359] Metadata directory not provided
I20260927 12:44:53.959772 20308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:44:53.959928 20308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:44:53.961964 20308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512840836255-392-0/minicluster-data/ts-1/data/instance:
uuid: "2e81f3bdbf9f4bd5987e4d804ea2af22"
format_stamp: "Formatted at 2026-09-27 12:44:53 on dist-test-slave-bnc1"
server_key: "905f8f60ef7feb0f05a7cea6df3bc7a0"
server_key_iv: "0c5aef36a9c3fbff9ed6c428bf16237c"
server_key_version: "encryptionkey@0"
I20260927 12:44:53.962471 20308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512840836255-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2e81f3bdbf9f4bd5987e4d804ea2af22"
format_stamp: "Formatted at 2026-09-27 12:44:53 on dist-test-slave-bnc1"
server_key: "905f8f60ef7feb0f05a7cea6df3bc7a0"
server_key_iv: "0c5aef36a9c3fbff9ed6c428bf16237c"
server_key_version: "encryptionkey@0"
I20260927 12:44:53.966073 20308 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 12:44:53.968421 20322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:44:53.969776 20308 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 12:44:53.969871 20308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512840836255-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512840836255-392-0/minicluster-data/ts-1/wal
uuid: "2e81f3bdbf9f4bd5987e4d804ea2af22"
format_stamp: "Formatted at 2026-09-27 12:44:53 on dist-test-slave-bnc1"
server_key: "905f8f60ef7feb0f05a7cea6df3bc7a0"
server_key_iv: "0c5aef36a9c3fbff9ed6c428bf16237c"
server_key_version: "encryptionkey@0"
I20260927 12:44:53.969956 20308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512840836255-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512840836255-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512840836255-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:44:53.991591 20308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:44:53.992267 20308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:44:53.992421 20308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:44:53.992983 20308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:44:53.993964 20308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:44:53.994058 20308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:44:53.994129 20308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:44:53.994185 20308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:44:54.003777 20308 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35637
I20260927 12:44:54.003820 20435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35637 every 8 connection(s)
I20260927 12:44:54.004704 20308 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512840836255-392-0/minicluster-data/ts-1/data/info.pb
I20260927 12:44:54.005749 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 20308
I20260927 12:44:54.005896 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512840836255-392-0/minicluster-data/ts-1/wal/instance
I20260927 12:44:54.006213 392 external_mini_cluster.cc:1428] Setting key ba75a54ac555c1252f8de48cf511ed8a
I20260927 12:44:54.008502 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512840836255-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512840836255-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512840836255-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40049
--builtin_ntp_servers=127.0.98.20:37733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 12:44:54.015990 20156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:54.007042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40819 (local address 127.0.98.62:40049)
0927 12:44:54.007210 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:54.007213 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:54.008203 (+ 990us) server_negotiation.cc:407] Connection header received
0927 12:44:54.010305 (+ 2102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:54.010310 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:54.010372 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:54.010470 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:44:54.011355 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:54.011932 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:54.013861 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:54.014070 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:54.014159 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:54.015486 (+ 1327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:44:54.015489 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:44:54.015491 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:44:54.015503 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:44:54.015521 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:44:54.015528 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:44:54.015585 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:54.015796 (+ 211us) server_negotiation.cc:299] Negotiation successful
0927 12:44:54.015844 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 12:44:54.017292 20438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:54.007219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40049 (local address 127.0.98.2:40819)
0927 12:44:54.008031 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:54.008087 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 12:44:54.010054 (+ 1967us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:44:54.010627 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:44:54.010634 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:44:54.010721 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:44:54.011222 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:54.011235 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:54.012069 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:54.012076 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:54.013733 (+ 1657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:54.013749 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:54.014233 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:54.014241 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:54.014490 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:54.015167 (+ 677us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 12:44:54.015171 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:44:54.015232 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 12:44:54.015323 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:44:54.015616 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:44:54.015628 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:44:54.015638 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 12:44:54.015798 (+ 160us) client_negotiation.cc:240] Negotiation successful
0927 12:44:54.016155 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":135,"threads_started":1}
I20260927 12:44:54.018502 20436 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40049
I20260927 12:44:54.018754 20436 heartbeater.cc:464] Registering TS with master...
I20260927 12:44:54.019269 20436 heartbeater.cc:511] Master 127.0.98.62:40049 requested a full tablet report, sending...
I20260927 12:44:54.020253 20117 ts_manager.cc:194] Registered new tserver with Master: 2e81f3bdbf9f4bd5987e4d804ea2af22 (127.0.98.2:35637)
I20260927 12:44:54.020834 20117 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:40819
W20260927 12:44:54.122785 20439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:44:54.123025 20439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:44:54.123092 20439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:44:54.127213 20439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:44:54.127279 20439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:44:54.127334 20439 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260927 12:44:54.132184 20439 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512840836255-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512840836255-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40049
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20439
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512840836255-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:44:54.133340 20439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:44:54.134827 20439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 12:44:54.144560 20439 server_base.cc:1034] running on GCE node
W20260927 12:44:54.144574 20444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:44:54.144574 20445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:44:54.144733 20447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:44:54.145316 20439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:44:54.146055 20439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:44:54.147254 20439 hybrid_clock.cc:648] HybridClock initialized: now 1790513094147237 us; error 28 us; skew 500 ppm
I20260927 12:44:54.149055 20439 webserver.cc:460] Webserver started at http://127.0.98.3:41187/ using document root <none> and password file <none>
I20260927 12:44:54.149720 20439 fs_manager.cc:359] Metadata directory not provided
I20260927 12:44:54.149813 20439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:44:54.150022 20439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:44:54.152064 20439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512840836255-392-0/minicluster-data/ts-2/data/instance:
uuid: "4333c8740df347c683c633ac69ebb77f"
format_stamp: "Formatted at 2026-09-27 12:44:54 on dist-test-slave-bnc1"
server_key: "6396f0c77a864d340fbcb3adfc1357c2"
server_key_iv: "c866ff92170448767f4bd9bd4fb26ea7"
server_key_version: "encryptionkey@0"
I20260927 12:44:54.152740 20439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512840836255-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4333c8740df347c683c633ac69ebb77f"
format_stamp: "Formatted at 2026-09-27 12:44:54 on dist-test-slave-bnc1"
server_key: "6396f0c77a864d340fbcb3adfc1357c2"
server_key_iv: "c866ff92170448767f4bd9bd4fb26ea7"
server_key_version: "encryptionkey@0"
I20260927 12:44:54.156472 20439 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260927 12:44:54.158968 20453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:44:54.160254 20439 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 12:44:54.160399 20439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512840836255-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512840836255-392-0/minicluster-data/ts-2/wal
uuid: "4333c8740df347c683c633ac69ebb77f"
format_stamp: "Formatted at 2026-09-27 12:44:54 on dist-test-slave-bnc1"
server_key: "6396f0c77a864d340fbcb3adfc1357c2"
server_key_iv: "c866ff92170448767f4bd9bd4fb26ea7"
server_key_version: "encryptionkey@0"
I20260927 12:44:54.160518 20439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512840836255-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512840836255-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512840836255-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:44:54.177732 20439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:44:54.178421 20439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:44:54.178611 20439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:44:54.179243 20439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 12:44:54.180181 20439 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 12:44:54.180282 20439 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 12:44:54.180352 20439 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 12:44:54.180388 20439 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 12:44:54.189275 20439 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42083
I20260927 12:44:54.189301 20566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42083 every 8 connection(s)
I20260927 12:44:54.190223 20439 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512840836255-392-0/minicluster-data/ts-2/data/info.pb
I20260927 12:44:54.195654 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 20439
I20260927 12:44:54.195783 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512840836255-392-0/minicluster-data/ts-2/wal/instance
I20260927 12:44:54.196127 392 external_mini_cluster.cc:1428] Setting key 49bcdaed50ac671e25969987d6397de8
I20260927 12:44:54.199540 20156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:54.192049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51769 (local address 127.0.98.62:40049)
0927 12:44:54.192242 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:54.192247 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:54.193247 (+ 1000us) server_negotiation.cc:407] Connection header received
0927 12:44:54.194746 (+ 1499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:54.194748 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:54.194794 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:54.194867 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:44:54.195826 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:54.196594 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:54.197799 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:54.198073 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:54.198221 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:54.199180 (+ 959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:44:54.199183 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:44:54.199185 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:44:54.199196 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:44:54.199215 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:44:54.199223 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:44:54.199280 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:54.199389 (+ 109us) server_negotiation.cc:299] Negotiation successful
0927 12:44:54.199436 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 12:44:54.200315 20569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:54.192473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40049 (local address 127.0.98.3:51769)
0927 12:44:54.193093 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:54.193132 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 12:44:54.194514 (+ 1382us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:44:54.195044 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:44:54.195051 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:44:54.195133 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:44:54.195620 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:54.195632 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:54.196756 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:54.196759 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:54.197679 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:54.197688 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:54.198238 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:54.198242 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:54.198424 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:54.198939 (+ 515us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 12:44:54.198941 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:44:54.198978 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 12:44:54.199035 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:44:54.199281 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:44:54.199286 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:44:54.199289 (+ 3us) client_negotiation.cc:769] Sending connection context
0927 12:44:54.199373 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 12:44:54.199566 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":110,"threads_started":1}
I20260927 12:44:54.201287 20567 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40049
I20260927 12:44:54.201509 20567 heartbeater.cc:464] Registering TS with master...
I20260927 12:44:54.201997 20567 heartbeater.cc:511] Master 127.0.98.62:40049 requested a full tablet report, sending...
I20260927 12:44:54.202940 20117 ts_manager.cc:194] Registered new tserver with Master: 4333c8740df347c683c633ac69ebb77f (127.0.98.3:42083)
I20260927 12:44:54.203495 20117 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:51769
I20260927 12:44:54.209830 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 12:44:54.215005 20156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:54.211596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39562 (local address 127.0.98.62:40049)
0927 12:44:54.211728 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:54.211732 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:54.211921 (+ 189us) server_negotiation.cc:407] Connection header received
0927 12:44:54.212013 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:54.212015 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:54.212053 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:54.212118 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:44:54.212356 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:54.212842 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:54.213626 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:54.213831 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:54.213908 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:54.214605 (+ 697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:44:54.214608 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:44:54.214610 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:44:54.214621 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:44:54.214637 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:44:54.214645 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:44:54.214703 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:54.214784 (+ 81us) server_negotiation.cc:299] Negotiation successful
0927 12:44:54.214828 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 12:44:54.217540 20117 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39562:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 12:44:54.220573 20117 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 12:44:54.235749 20581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:54.229618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52000 (local address 127.0.98.1:44097)
0927 12:44:54.230039 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:54.230051 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:54.230267 (+ 216us) server_negotiation.cc:407] Connection header received
0927 12:44:54.230422 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:54.230429 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:54.230676 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:54.231210 (+ 534us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:44:54.231354 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:54.232579 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:54.234227 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:54.235030 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:54.235150 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:54.235372 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:54.235462 (+ 90us) server_negotiation.cc:299] Negotiation successful
0927 12:44:54.235565 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":123,"threads_started":1}
I20260927 12:44:54.235742 20579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:54.229395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44097 (local address 127.0.0.1:52000)
0927 12:44:54.230093 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:54.230157 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 12:44:54.230283 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:44:54.230853 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:44:54.230858 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:44:54.230882 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:44:54.231054 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:54.231062 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:54.232710 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:54.232714 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:54.234085 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:54.234094 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:54.235159 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:54.235162 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:54.235270 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:54.235321 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 12:44:54.235431 (+ 110us) client_negotiation.cc:240] Negotiation successful
0927 12:44:54.235559 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"mutex_wait_us":126,"thread_start_us":132,"threads_started":1}
I20260927 12:44:54.235915 20582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:54.230243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54854 (local address 127.0.98.2:35637)
0927 12:44:54.230574 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:54.230585 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:54.230607 (+ 22us) server_negotiation.cc:407] Connection header received
0927 12:44:54.230685 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:54.230691 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:54.230861 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:54.231015 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:44:54.231404 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:54.232579 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:54.234443 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:54.235199 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:54.235300 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:54.235451 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:54.235636 (+ 185us) server_negotiation.cc:299] Negotiation successful
0927 12:44:54.235737 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":115,"threads_started":1}
I20260927 12:44:54.236227 20578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:54.229261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35637 (local address 127.0.0.1:54854)
0927 12:44:54.230092 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:54.230146 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 12:44:54.230274 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:44:54.231172 (+ 898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:44:54.231175 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:44:54.231191 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:44:54.231279 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:54.231285 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:54.232740 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:54.232743 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:54.234317 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:54.234322 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:54.235320 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:54.235323 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:54.235411 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:54.235438 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 12:44:54.235502 (+ 64us) client_negotiation.cc:240] Negotiation successful
0927 12:44:54.235558 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"mutex_wait_us":217,"thread_start_us":86,"threads_started":1}
I20260927 12:44:54.236775 20580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:54.229661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42083 (local address 127.0.0.1:35718)
0927 12:44:54.230760 (+ 1099us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:54.230775 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:44:54.230863 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:44:54.231615 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:44:54.231618 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:44:54.231636 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:44:54.231744 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:54.231753 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:54.232805 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:54.232808 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:54.234308 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:54.234320 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:54.236451 (+ 2131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:54.236453 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:54.236522 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:54.236541 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 12:44:54.236592 (+ 51us) client_negotiation.cc:240] Negotiation successful
0927 12:44:54.236656 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1040,"mutex_wait_us":119,"thread_start_us":82,"threads_started":1}
I20260927 12:44:54.236914 20583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:54.230527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35718 (local address 127.0.98.3:42083)
0927 12:44:54.231119 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:54.231131 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:54.231168 (+ 37us) server_negotiation.cc:407] Connection header received
0927 12:44:54.231248 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:54.231256 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:54.231461 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:54.231639 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:44:54.231877 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:54.232618 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:54.235525 (+ 2907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:54.236341 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:54.236450 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:54.236612 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:54.236656 (+ 44us) server_negotiation.cc:299] Negotiation successful
0927 12:44:54.236763 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":124,"threads_started":1}
I20260927 12:44:54.239415 20501 tablet_service.cc:1467] Processing CreateTablet for tablet 65adafab8ef9417a9151d15ebd693a1e (DEFAULT_TABLE table=test-table [id=4106d860b540420d84a8178826d84376]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:44:54.239430 20370 tablet_service.cc:1467] Processing CreateTablet for tablet 65adafab8ef9417a9151d15ebd693a1e (DEFAULT_TABLE table=test-table [id=4106d860b540420d84a8178826d84376]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:44:54.239430 20239 tablet_service.cc:1467] Processing CreateTablet for tablet 65adafab8ef9417a9151d15ebd693a1e (DEFAULT_TABLE table=test-table [id=4106d860b540420d84a8178826d84376]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 12:44:54.240434 20501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65adafab8ef9417a9151d15ebd693a1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:44:54.240439 20239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65adafab8ef9417a9151d15ebd693a1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:44:54.240439 20370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65adafab8ef9417a9151d15ebd693a1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:44:54.246363 20584 tablet_bootstrap.cc:492] T 65adafab8ef9417a9151d15ebd693a1e P 4333c8740df347c683c633ac69ebb77f: Bootstrap starting.
I20260927 12:44:54.247977 20585 tablet_bootstrap.cc:492] T 65adafab8ef9417a9151d15ebd693a1e P 2e81f3bdbf9f4bd5987e4d804ea2af22: Bootstrap starting.
I20260927 12:44:54.248678 20584 tablet_bootstrap.cc:654] T 65adafab8ef9417a9151d15ebd693a1e P 4333c8740df347c683c633ac69ebb77f: Neither blocks nor log segments found. Creating new log.
I20260927 12:44:54.249493 20584 log.cc:824] T 65adafab8ef9417a9151d15ebd693a1e P 4333c8740df347c683c633ac69ebb77f: Log is configured to *not* fsync() on all Append() calls
I20260927 12:44:54.249886 20585 tablet_bootstrap.cc:654] T 65adafab8ef9417a9151d15ebd693a1e P 2e81f3bdbf9f4bd5987e4d804ea2af22: Neither blocks nor log segments found. Creating new log.
I20260927 12:44:54.250201 20586 tablet_bootstrap.cc:492] T 65adafab8ef9417a9151d15ebd693a1e P 14d7a46b190546d59b14725d3c378176: Bootstrap starting.
I20260927 12:44:54.250694 20585 log.cc:824] T 65adafab8ef9417a9151d15ebd693a1e P 2e81f3bdbf9f4bd5987e4d804ea2af22: Log is configured to *not* fsync() on all Append() calls
I20260927 12:44:54.251636 20584 tablet_bootstrap.cc:492] T 65adafab8ef9417a9151d15ebd693a1e P 4333c8740df347c683c633ac69ebb77f: No bootstrap required, opened a new log
I20260927 12:44:54.251821 20584 ts_tablet_manager.cc:1397] T 65adafab8ef9417a9151d15ebd693a1e P 4333c8740df347c683c633ac69ebb77f: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20260927 12:44:54.252053 20586 tablet_bootstrap.cc:654] T 65adafab8ef9417a9151d15ebd693a1e P 14d7a46b190546d59b14725d3c378176: Neither blocks nor log segments found. Creating new log.
I20260927 12:44:54.252671 20586 log.cc:824] T 65adafab8ef9417a9151d15ebd693a1e P 14d7a46b190546d59b14725d3c378176: Log is configured to *not* fsync() on all Append() calls
I20260927 12:44:54.252700 20585 tablet_bootstrap.cc:492] T 65adafab8ef9417a9151d15ebd693a1e P 2e81f3bdbf9f4bd5987e4d804ea2af22: No bootstrap required, opened a new log
I20260927 12:44:54.252874 20585 ts_tablet_manager.cc:1397] T 65adafab8ef9417a9151d15ebd693a1e P 2e81f3bdbf9f4bd5987e4d804ea2af22: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260927 12:44:54.255127 20586 tablet_bootstrap.cc:492] T 65adafab8ef9417a9151d15ebd693a1e P 14d7a46b190546d59b14725d3c378176: No bootstrap required, opened a new log
I20260927 12:44:54.255327 20586 ts_tablet_manager.cc:1397] T 65adafab8ef9417a9151d15ebd693a1e P 14d7a46b190546d59b14725d3c378176: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260927 12:44:54.256263 20584 raft_consensus.cc:348] T 65adafab8ef9417a9151d15ebd693a1e P 4333c8740df347c683c633ac69ebb77f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e81f3bdbf9f4bd5987e4d804ea2af22" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35637 } } peers { permanent_uuid: "14d7a46b190546d59b14725d3c378176" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44097 } } peers { permanent_uuid: "4333c8740df347c683c633ac69ebb77f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42083 } }
I20260927 12:44:54.256546 20584 raft_consensus.cc:374] T 65adafab8ef9417a9151d15ebd693a1e P 4333c8740df347c683c633ac69ebb77f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:44:54.256603 20584 raft_consensus.cc:729] T 65adafab8ef9417a9151d15ebd693a1e P 4333c8740df347c683c633ac69ebb77f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4333c8740df347c683c633ac69ebb77f, State: Initialized, Role: FOLLOWER
I20260927 12:44:54.257053 20584 consensus_queue.cc:260] T 65adafab8ef9417a9151d15ebd693a1e P 4333c8740df347c683c633ac69ebb77f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e81f3bdbf9f4bd5987e4d804ea2af22" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35637 } } peers { permanent_uuid: "14d7a46b190546d59b14725d3c378176" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44097 } } peers { permanent_uuid: "4333c8740df347c683c633ac69ebb77f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42083 } }
I20260927 12:44:54.257004 20585 raft_consensus.cc:348] T 65adafab8ef9417a9151d15ebd693a1e P 2e81f3bdbf9f4bd5987e4d804ea2af22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e81f3bdbf9f4bd5987e4d804ea2af22" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35637 } } peers { permanent_uuid: "14d7a46b190546d59b14725d3c378176" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44097 } } peers { permanent_uuid: "4333c8740df347c683c633ac69ebb77f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42083 } }
I20260927 12:44:54.257228 20585 raft_consensus.cc:374] T 65adafab8ef9417a9151d15ebd693a1e P 2e81f3bdbf9f4bd5987e4d804ea2af22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:44:54.257284 20585 raft_consensus.cc:729] T 65adafab8ef9417a9151d15ebd693a1e P 2e81f3bdbf9f4bd5987e4d804ea2af22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e81f3bdbf9f4bd5987e4d804ea2af22, State: Initialized, Role: FOLLOWER
I20260927 12:44:54.257736 20585 consensus_queue.cc:260] T 65adafab8ef9417a9151d15ebd693a1e P 2e81f3bdbf9f4bd5987e4d804ea2af22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e81f3bdbf9f4bd5987e4d804ea2af22" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35637 } } peers { permanent_uuid: "14d7a46b190546d59b14725d3c378176" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44097 } } peers { permanent_uuid: "4333c8740df347c683c633ac69ebb77f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42083 } }
I20260927 12:44:54.258126 20584 ts_tablet_manager.cc:1428] T 65adafab8ef9417a9151d15ebd693a1e P 4333c8740df347c683c633ac69ebb77f: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260927 12:44:54.258455 20567 heartbeater.cc:503] Master 127.0.98.62:40049 was elected leader, sending a full tablet report...
I20260927 12:44:54.258457 20436 heartbeater.cc:503] Master 127.0.98.62:40049 was elected leader, sending a full tablet report...
I20260927 12:44:54.258682 20585 ts_tablet_manager.cc:1428] T 65adafab8ef9417a9151d15ebd693a1e P 2e81f3bdbf9f4bd5987e4d804ea2af22: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260927 12:44:54.258741 20586 raft_consensus.cc:348] T 65adafab8ef9417a9151d15ebd693a1e P 14d7a46b190546d59b14725d3c378176 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e81f3bdbf9f4bd5987e4d804ea2af22" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35637 } } peers { permanent_uuid: "14d7a46b190546d59b14725d3c378176" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44097 } } peers { permanent_uuid: "4333c8740df347c683c633ac69ebb77f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42083 } }
I20260927 12:44:54.258961 20586 raft_consensus.cc:374] T 65adafab8ef9417a9151d15ebd693a1e P 14d7a46b190546d59b14725d3c378176 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:44:54.259004 20586 raft_consensus.cc:729] T 65adafab8ef9417a9151d15ebd693a1e P 14d7a46b190546d59b14725d3c378176 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14d7a46b190546d59b14725d3c378176, State: Initialized, Role: FOLLOWER
I20260927 12:44:54.259711 20586 consensus_queue.cc:260] T 65adafab8ef9417a9151d15ebd693a1e P 14d7a46b190546d59b14725d3c378176 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e81f3bdbf9f4bd5987e4d804ea2af22" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35637 } } peers { permanent_uuid: "14d7a46b190546d59b14725d3c378176" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44097 } } peers { permanent_uuid: "4333c8740df347c683c633ac69ebb77f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42083 } }
I20260927 12:44:54.260473 20305 heartbeater.cc:503] Master 127.0.98.62:40049 was elected leader, sending a full tablet report...
I20260927 12:44:54.260625 20586 ts_tablet_manager.cc:1428] T 65adafab8ef9417a9151d15ebd693a1e P 14d7a46b190546d59b14725d3c378176: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260927 12:44:54.316956 20306 tablet.cc:2369] T 65adafab8ef9417a9151d15ebd693a1e P 14d7a46b190546d59b14725d3c378176: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:44:54.329931 20592 raft_consensus.cc:482] T 65adafab8ef9417a9151d15ebd693a1e P 14d7a46b190546d59b14725d3c378176 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 12:44:54.330188 20592 raft_consensus.cc:504] T 65adafab8ef9417a9151d15ebd693a1e P 14d7a46b190546d59b14725d3c378176 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e81f3bdbf9f4bd5987e4d804ea2af22" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35637 } } peers { permanent_uuid: "14d7a46b190546d59b14725d3c378176" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44097 } } peers { permanent_uuid: "4333c8740df347c683c633ac69ebb77f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42083 } }
I20260927 12:44:54.331413 20592 leader_election.cc:288] T 65adafab8ef9417a9151d15ebd693a1e P 14d7a46b190546d59b14725d3c378176 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2e81f3bdbf9f4bd5987e4d804ea2af22 (127.0.98.2:35637), 4333c8740df347c683c633ac69ebb77f (127.0.98.3:42083)
I20260927 12:44:54.335443 20583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:54.331790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53595 (local address 127.0.98.3:42083)
0927 12:44:54.332000 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:54.332005 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:54.332181 (+ 176us) server_negotiation.cc:407] Connection header received
0927 12:44:54.332344 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:54.332352 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:54.332392 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:54.332482 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:44:54.332749 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:54.333242 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:54.334449 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:54.334890 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:54.335012 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:54.335071 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:54.335255 (+ 184us) server_negotiation.cc:299] Negotiation successful
0927 12:44:54.335320 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260927 12:44:54.335460 20594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:54.331699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42083 (local address 127.0.98.1:53595)
0927 12:44:54.332090 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:54.332101 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0927 12:44:54.332224 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:44:54.332492 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:44:54.332495 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:44:54.332511 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:44:54.332635 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:54.332640 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:54.333361 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:54.333367 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:54.334319 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:54.334329 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:54.335017 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:54.335021 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:54.335116 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:54.335139 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 12:44:54.335239 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 12:44:54.335310 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":106,"threads_started":1}
I20260927 12:44:54.335565 20582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:54.331845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60193 (local address 127.0.98.2:35637)
0927 12:44:54.331981 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:54.331987 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:54.332083 (+ 96us) server_negotiation.cc:407] Connection header received
0927 12:44:54.332265 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:54.332267 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:54.332322 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:54.332405 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 12:44:54.332683 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:54.333196 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:54.334629 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:54.335024 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:54.335172 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:54.335234 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:54.335362 (+ 128us) server_negotiation.cc:299] Negotiation successful
0927 12:44:54.335425 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 12:44:54.335799 20593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:54.331686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35637 (local address 127.0.98.1:60193)
0927 12:44:54.331963 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:54.331980 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 12:44:54.332149 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:44:54.332417 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:44:54.332420 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:44:54.332436 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 12:44:54.332564 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:54.332569 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:54.333311 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:54.333317 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:54.334497 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:44:54.334509 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:44:54.335145 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:44:54.335148 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:44:54.335241 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:54.335267 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 12:44:54.335342 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 12:44:54.335412 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":54,"thread_start_us":73,"threads_started":1}
I20260927 12:44:54.336083 20521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65adafab8ef9417a9151d15ebd693a1e" candidate_uuid: "14d7a46b190546d59b14725d3c378176" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4333c8740df347c683c633ac69ebb77f" is_pre_election: true
I20260927 12:44:54.336434 20521 raft_consensus.cc:2454] T 65adafab8ef9417a9151d15ebd693a1e P 4333c8740df347c683c633ac69ebb77f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14d7a46b190546d59b14725d3c378176 in term 0.
I20260927 12:44:54.336442 20390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65adafab8ef9417a9151d15ebd693a1e" candidate_uuid: "14d7a46b190546d59b14725d3c378176" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e81f3bdbf9f4bd5987e4d804ea2af22" is_pre_election: true
I20260927 12:44:54.336712 20390 raft_consensus.cc:2454] T 65adafab8ef9417a9151d15ebd693a1e P 2e81f3bdbf9f4bd5987e4d804ea2af22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14d7a46b190546d59b14725d3c378176 in term 0.
I20260927 12:44:54.336987 20195 leader_election.cc:302] T 65adafab8ef9417a9151d15ebd693a1e P 14d7a46b190546d59b14725d3c378176 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14d7a46b190546d59b14725d3c378176, 4333c8740df347c683c633ac69ebb77f; no voters:
I20260927 12:44:54.337301 20592 raft_consensus.cc:2781] T 65adafab8ef9417a9151d15ebd693a1e P 14d7a46b190546d59b14725d3c378176 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 12:44:54.337387 20592 raft_consensus.cc:482] T 65adafab8ef9417a9151d15ebd693a1e P 14d7a46b190546d59b14725d3c378176 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 12:44:54.337430 20592 raft_consensus.cc:3037] T 65adafab8ef9417a9151d15ebd693a1e P 14d7a46b190546d59b14725d3c378176 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:44:54.338474 20592 raft_consensus.cc:504] T 65adafab8ef9417a9151d15ebd693a1e P 14d7a46b190546d59b14725d3c378176 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e81f3bdbf9f4bd5987e4d804ea2af22" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35637 } } peers { permanent_uuid: "14d7a46b190546d59b14725d3c378176" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44097 } } peers { permanent_uuid: "4333c8740df347c683c633ac69ebb77f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42083 } }
I20260927 12:44:54.338856 20592 leader_election.cc:288] T 65adafab8ef9417a9151d15ebd693a1e P 14d7a46b190546d59b14725d3c378176 [CANDIDATE]: Term 1 election: Requested vote from peers 2e81f3bdbf9f4bd5987e4d804ea2af22 (127.0.98.2:35637), 4333c8740df347c683c633ac69ebb77f (127.0.98.3:42083)
I20260927 12:44:54.339148 20390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65adafab8ef9417a9151d15ebd693a1e" candidate_uuid: "14d7a46b190546d59b14725d3c378176" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e81f3bdbf9f4bd5987e4d804ea2af22"
I20260927 12:44:54.339234 20521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65adafab8ef9417a9151d15ebd693a1e" candidate_uuid: "14d7a46b190546d59b14725d3c378176" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4333c8740df347c683c633ac69ebb77f"
I20260927 12:44:54.339308 20390 raft_consensus.cc:3037] T 65adafab8ef9417a9151d15ebd693a1e P 2e81f3bdbf9f4bd5987e4d804ea2af22 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:44:54.339372 20521 raft_consensus.cc:3037] T 65adafab8ef9417a9151d15ebd693a1e P 4333c8740df347c683c633ac69ebb77f [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:44:54.340451 20521 raft_consensus.cc:2454] T 65adafab8ef9417a9151d15ebd693a1e P 4333c8740df347c683c633ac69ebb77f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14d7a46b190546d59b14725d3c378176 in term 1.
I20260927 12:44:54.340534 20390 raft_consensus.cc:2454] T 65adafab8ef9417a9151d15ebd693a1e P 2e81f3bdbf9f4bd5987e4d804ea2af22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14d7a46b190546d59b14725d3c378176 in term 1.
I20260927 12:44:54.340857 20193 leader_election.cc:302] T 65adafab8ef9417a9151d15ebd693a1e P 14d7a46b190546d59b14725d3c378176 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14d7a46b190546d59b14725d3c378176, 2e81f3bdbf9f4bd5987e4d804ea2af22; no voters:
I20260927 12:44:54.341107 20592 raft_consensus.cc:2781] T 65adafab8ef9417a9151d15ebd693a1e P 14d7a46b190546d59b14725d3c378176 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:44:54.341336 20592 raft_consensus.cc:686] T 65adafab8ef9417a9151d15ebd693a1e P 14d7a46b190546d59b14725d3c378176 [term 1 LEADER]: Becoming Leader. State: Replica: 14d7a46b190546d59b14725d3c378176, State: Running, Role: LEADER
I20260927 12:44:54.341724 20592 consensus_queue.cc:237] T 65adafab8ef9417a9151d15ebd693a1e P 14d7a46b190546d59b14725d3c378176 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e81f3bdbf9f4bd5987e4d804ea2af22" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35637 } } peers { permanent_uuid: "14d7a46b190546d59b14725d3c378176" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44097 } } peers { permanent_uuid: "4333c8740df347c683c633ac69ebb77f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42083 } }
I20260927 12:44:54.345027 20116 catalog_manager.cc:5495] T 65adafab8ef9417a9151d15ebd693a1e P 14d7a46b190546d59b14725d3c378176 reported cstate change: term changed from 0 to 1, leader changed from <none> to 14d7a46b190546d59b14725d3c378176 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "14d7a46b190546d59b14725d3c378176" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e81f3bdbf9f4bd5987e4d804ea2af22" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35637 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14d7a46b190546d59b14725d3c378176" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4333c8740df347c683c633ac69ebb77f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42083 } health_report { overall_health: UNKNOWN } } }
I20260927 12:44:54.366523 20581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:54.363064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52016 (local address 127.0.98.1:44097)
0927 12:44:54.363192 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:54.363196 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:54.363209 (+ 13us) server_negotiation.cc:407] Connection header received
0927 12:44:54.363306 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:54.363308 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:54.363364 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:54.363446 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:44:54.363731 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:54.364239 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:54.365131 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:54.365375 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:54.365458 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:54.365787 (+ 329us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:44:54.366212 (+ 425us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:44:54.366307 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:54.366359 (+ 52us) server_negotiation.cc:299] Negotiation successful
0927 12:44:54.366405 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 12:44:54.372351 20390 raft_consensus.cc:1261] T 65adafab8ef9417a9151d15ebd693a1e P 2e81f3bdbf9f4bd5987e4d804ea2af22 [term 1 FOLLOWER]: Refusing update from remote peer 14d7a46b190546d59b14725d3c378176: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:44:54.372350 20521 raft_consensus.cc:1261] T 65adafab8ef9417a9151d15ebd693a1e P 4333c8740df347c683c633ac69ebb77f [term 1 FOLLOWER]: Refusing update from remote peer 14d7a46b190546d59b14725d3c378176: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:44:54.373104 20595 consensus_queue.cc:1035] T 65adafab8ef9417a9151d15ebd693a1e P 14d7a46b190546d59b14725d3c378176 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e81f3bdbf9f4bd5987e4d804ea2af22" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:44:54.373307 20592 consensus_queue.cc:1035] T 65adafab8ef9417a9151d15ebd693a1e P 14d7a46b190546d59b14725d3c378176 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4333c8740df347c683c633ac69ebb77f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:44:54.383180 20600 mvcc.cc:205] Tried to move back new op lower bound from 7333941634541891584 to 7333941634426195968. Current Snapshot: MvccSnapshot[applied={T|T < 7333941634541891584}]
I20260927 12:44:54.385421 20599 mvcc.cc:205] Tried to move back new op lower bound from 7333941634541891584 to 7333941634426195968. Current Snapshot: MvccSnapshot[applied={T|T < 7333941634541891584 or (T in {7333941634541891584})}]
I20260927 12:44:54.389509 20117 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:44:54.389732 20117 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 12:44:54.392174 20117 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f678ce7941b44d3a984688043d96e789: Sending DeleteTablet for 3 replicas of tablet 65adafab8ef9417a9151d15ebd693a1e
I20260927 12:44:54.392978 20239 tablet_service.cc:1514] Processing DeleteTablet for tablet 65adafab8ef9417a9151d15ebd693a1e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:44:54 UTC) from {username='slave'} at 127.0.0.1:52000
I20260927 12:44:54.392978 20370 tablet_service.cc:1514] Processing DeleteTablet for tablet 65adafab8ef9417a9151d15ebd693a1e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:44:54 UTC) from {username='slave'} at 127.0.0.1:54854
I20260927 12:44:54.393153 20501 tablet_service.cc:1514] Processing DeleteTablet for tablet 65adafab8ef9417a9151d15ebd693a1e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 12:44:54 UTC) from {username='slave'} at 127.0.0.1:35718
I20260927 12:44:54.393467 20608 tablet_replica.cc:330] T 65adafab8ef9417a9151d15ebd693a1e P 14d7a46b190546d59b14725d3c378176: stopping tablet replica
I20260927 12:44:54.393671 20607 tablet_replica.cc:330] T 65adafab8ef9417a9151d15ebd693a1e P 2e81f3bdbf9f4bd5987e4d804ea2af22: stopping tablet replica
I20260927 12:44:54.393966 20609 tablet_replica.cc:330] T 65adafab8ef9417a9151d15ebd693a1e P 4333c8740df347c683c633ac69ebb77f: stopping tablet replica
I20260927 12:44:54.393981 20607 raft_consensus.cc:2229] T 65adafab8ef9417a9151d15ebd693a1e P 2e81f3bdbf9f4bd5987e4d804ea2af22 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:44:54.394254 20609 raft_consensus.cc:2229] T 65adafab8ef9417a9151d15ebd693a1e P 4333c8740df347c683c633ac69ebb77f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 12:44:54.394382 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 20177
I20260927 12:44:54.394524 20607 raft_consensus.cc:2258] T 65adafab8ef9417a9151d15ebd693a1e P 2e81f3bdbf9f4bd5987e4d804ea2af22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:44:54.394547 20609 raft_consensus.cc:2258] T 65adafab8ef9417a9151d15ebd693a1e P 4333c8740df347c683c633ac69ebb77f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 12:44:54.396163 20607 ts_tablet_manager.cc:1905] T 65adafab8ef9417a9151d15ebd693a1e P 2e81f3bdbf9f4bd5987e4d804ea2af22: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:44:54.397220 20609 ts_tablet_manager.cc:1905] T 65adafab8ef9417a9151d15ebd693a1e P 4333c8740df347c683c633ac69ebb77f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 12:44:54.398792 20607 ts_tablet_manager.cc:1918] T 65adafab8ef9417a9151d15ebd693a1e P 2e81f3bdbf9f4bd5987e4d804ea2af22: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:44:54.398907 20607 log.cc:1196] T 65adafab8ef9417a9151d15ebd693a1e P 2e81f3bdbf9f4bd5987e4d804ea2af22: Deleting WAL directory at /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512840836255-392-0/minicluster-data/ts-1/wal/wals/65adafab8ef9417a9151d15ebd693a1e
I20260927 12:44:54.399186 20607 ts_tablet_manager.cc:1939] T 65adafab8ef9417a9151d15ebd693a1e P 2e81f3bdbf9f4bd5987e4d804ea2af22: Deleting consensus metadata
I20260927 12:44:54.400324 20102 catalog_manager.cc:4841] TS 2e81f3bdbf9f4bd5987e4d804ea2af22 (127.0.98.2:35637): tablet 65adafab8ef9417a9151d15ebd693a1e (table test-table [id=4106d860b540420d84a8178826d84376]) successfully deleted
I20260927 12:44:54.400563 20609 ts_tablet_manager.cc:1918] T 65adafab8ef9417a9151d15ebd693a1e P 4333c8740df347c683c633ac69ebb77f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 12:44:54.400686 20609 log.cc:1196] T 65adafab8ef9417a9151d15ebd693a1e P 4333c8740df347c683c633ac69ebb77f: Deleting WAL directory at /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790512840836255-392-0/minicluster-data/ts-2/wal/wals/65adafab8ef9417a9151d15ebd693a1e
I20260927 12:44:54.401015 20609 ts_tablet_manager.cc:1939] T 65adafab8ef9417a9151d15ebd693a1e P 4333c8740df347c683c633ac69ebb77f: Deleting consensus metadata
W20260927 12:44:54.402119 20101 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44097 (error 108)
I20260927 12:44:54.402256 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 20308
I20260927 12:44:54.402232 20104 catalog_manager.cc:4841] TS 4333c8740df347c683c633ac69ebb77f (127.0.98.3:42083): tablet 65adafab8ef9417a9151d15ebd693a1e (table test-table [id=4106d860b540420d84a8178826d84376]) successfully deleted
I20260927 12:44:54.403923 20580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:54.403562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44097 (local address 127.0.0.1:52026)
0927 12:44:54.403733 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 12:44:54.403816 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 12:44:54.404136 20101 catalog_manager.cc:4584] TS 14d7a46b190546d59b14725d3c378176 (127.0.98.1:44097): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 65adafab8ef9417a9151d15ebd693a1e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44097: connect: Connection refused (error 111)
I20260927 12:44:54.408116 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 20439
I20260927 12:44:54.413704 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 20086
2026-09-27T12:44:54Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1051 ms)
[----------] 6 tests from EncryptionPolicyTest (10398 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790512840836255-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:44:54 dist-test-slave-bnc1 krb5kdc[20613](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:44:54 dist-test-slave-bnc1 krb5kdc[20613](info): set up 2 sockets
Sep 27 12:44:54 dist-test-slave-bnc1 krb5kdc[20613](info): commencing operation
krb5kdc: starting...
W20260927 12:44:55.457780 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790512840836255-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790512840836255-392-0/krb5kdc/test-user.keytab.
Sep 27 12:44:55 dist-test-slave-bnc1 krb5kdc[20613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:44:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:44:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790512840836255-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790512840836255-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 12:44:55.576678 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790512840836255-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790512840836255-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790512840836255-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46077
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790512840836255-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46077
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:44:55.686676 20626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:44:55.686967 20626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:44:55.687043 20626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:44:55.690969 20626 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:44:55.691056 20626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:44:55.691097 20626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:44:55.691164 20626 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:44:55.691200 20626 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:44:55.696365 20626 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790512840836255-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790512840836255-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46077
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790512840836255-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20626
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790512840836255-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:44:55.697433 20626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:44:55.698582 20626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:44:55.706902 20631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:44:55.706902 20634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:44:55.706928 20632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:44:55.707338 20626 server_base.cc:1034] running on GCE node
I20260927 12:44:55.707890 20626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:44:55.708940 20626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:44:55.710121 20626 hybrid_clock.cc:648] HybridClock initialized: now 1790513095710093 us; error 41 us; skew 500 ppm
Sep 27 12:44:55 dist-test-slave-bnc1 krb5kdc[20613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:44:55.712765 20626 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 12:44:55.713724 20626 webserver.cc:460] Webserver started at http://127.0.0.1:33661/ using document root <none> and password file <none>
I20260927 12:44:55.714263 20626 fs_manager.cc:359] Metadata directory not provided
I20260927 12:44:55.714351 20626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:44:55.714573 20626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:44:55.716495 20626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790512840836255-392-0/minicluster-data/master-0/data/instance:
uuid: "1edafcb6b9124e98903bde58d8b3424f"
format_stamp: "Formatted at 2026-09-27 12:44:55 on dist-test-slave-bnc1"
server_key: "e3b9be20be06f803e168630ee3fd5383"
server_key_iv: "e5539e5d424133e5fe62c9d434c219d8"
server_key_version: "encryptionkey@0"
I20260927 12:44:55.717033 20626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790512840836255-392-0/minicluster-data/master-0/wal/instance:
uuid: "1edafcb6b9124e98903bde58d8b3424f"
format_stamp: "Formatted at 2026-09-27 12:44:55 on dist-test-slave-bnc1"
server_key: "e3b9be20be06f803e168630ee3fd5383"
server_key_iv: "e5539e5d424133e5fe62c9d434c219d8"
server_key_version: "encryptionkey@0"
I20260927 12:44:55.720571 20626 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260927 12:44:55.722937 20641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:44:55.724109 20626 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 12:44:55.724253 20626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790512840836255-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790512840836255-392-0/minicluster-data/master-0/wal
uuid: "1edafcb6b9124e98903bde58d8b3424f"
format_stamp: "Formatted at 2026-09-27 12:44:55 on dist-test-slave-bnc1"
server_key: "e3b9be20be06f803e168630ee3fd5383"
server_key_iv: "e5539e5d424133e5fe62c9d434c219d8"
server_key_version: "encryptionkey@0"
I20260927 12:44:55.724362 20626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790512840836255-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790512840836255-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790512840836255-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:44:55.744625 20626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:44:55.747498 20626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:44:55.747697 20626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:44:55.755263 20626 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46077
I20260927 12:44:55.755291 20693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46077 every 8 connection(s)
I20260927 12:44:55.756253 20626 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:44:55.758991 20694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:44:55.762773 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 20626
I20260927 12:44:55.762882 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790512840836255-392-0/minicluster-data/master-0/wal/instance
I20260927 12:44:55.763293 392 external_mini_cluster.cc:1428] Setting key c993940a942cd229cb424924c9d779a9
I20260927 12:44:55.765015 20694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1edafcb6b9124e98903bde58d8b3424f: Bootstrap starting.
I20260927 12:44:55.767532 20694 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1edafcb6b9124e98903bde58d8b3424f: Neither blocks nor log segments found. Creating new log.
I20260927 12:44:55.768265 20694 log.cc:824] T 00000000000000000000000000000000 P 1edafcb6b9124e98903bde58d8b3424f: Log is configured to *not* fsync() on all Append() calls
I20260927 12:44:55.770112 20694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1edafcb6b9124e98903bde58d8b3424f: No bootstrap required, opened a new log
Sep 27 12:44:55 dist-test-slave-bnc1 krb5kdc[20613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 12:44:55.774906 20694 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1edafcb6b9124e98903bde58d8b3424f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1edafcb6b9124e98903bde58d8b3424f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46077 } }
I20260927 12:44:55.775198 20694 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1edafcb6b9124e98903bde58d8b3424f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:44:55.775261 20694 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1edafcb6b9124e98903bde58d8b3424f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1edafcb6b9124e98903bde58d8b3424f, State: Initialized, Role: FOLLOWER
I20260927 12:44:55.775705 20694 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1edafcb6b9124e98903bde58d8b3424f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1edafcb6b9124e98903bde58d8b3424f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46077 } }
I20260927 12:44:55.775867 20694 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1edafcb6b9124e98903bde58d8b3424f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:44:55.775949 20694 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1edafcb6b9124e98903bde58d8b3424f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:44:55.776062 20694 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1edafcb6b9124e98903bde58d8b3424f [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:44:55.777218 20694 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1edafcb6b9124e98903bde58d8b3424f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1edafcb6b9124e98903bde58d8b3424f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46077 } }
I20260927 12:44:55.777603 20694 leader_election.cc:302] T 00000000000000000000000000000000 P 1edafcb6b9124e98903bde58d8b3424f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1edafcb6b9124e98903bde58d8b3424f; no voters:
I20260927 12:44:55.777933 20694 leader_election.cc:288] T 00000000000000000000000000000000 P 1edafcb6b9124e98903bde58d8b3424f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:44:55.778046 20699 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1edafcb6b9124e98903bde58d8b3424f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:44:55.778287 20699 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1edafcb6b9124e98903bde58d8b3424f [term 1 LEADER]: Becoming Leader. State: Replica: 1edafcb6b9124e98903bde58d8b3424f, State: Running, Role: LEADER
I20260927 12:44:55.778594 20699 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1edafcb6b9124e98903bde58d8b3424f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1edafcb6b9124e98903bde58d8b3424f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46077 } }
I20260927 12:44:55.778750 20697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:55.764719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39325 (local address 127.0.0.1:46077)
0927 12:44:55.765182 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:55.765211 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:55.765250 (+ 39us) server_negotiation.cc:407] Connection header received
0927 12:44:55.765987 (+ 737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:55.766007 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:55.766361 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:55.766662 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:44:55.767555 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:55.768720 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:55.769689 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:55.770190 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:55.770393 (+ 203us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:55.773274 (+ 2881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:44:55.773282 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:44:55.773303 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:44:55.773359 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:44:55.775316 (+ 1957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:44:55.775861 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:44:55.775868 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:44:55.775887 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:44:55.776046 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:44:55.776411 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:44:55.776414 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:44:55.776426 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:44:55.776945 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:44:55.777167 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:55.777431 (+ 264us) server_negotiation.cc:299] Negotiation successful
0927 12:44:55.777689 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":148,"threads_started":1}
I20260927 12:44:55.779203 20694 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1edafcb6b9124e98903bde58d8b3424f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:44:55.780300 20701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1edafcb6b9124e98903bde58d8b3424f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1edafcb6b9124e98903bde58d8b3424f. Latest consensus state: current_term: 1 leader_uuid: "1edafcb6b9124e98903bde58d8b3424f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1edafcb6b9124e98903bde58d8b3424f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46077 } } }
I20260927 12:44:55.780444 20701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1edafcb6b9124e98903bde58d8b3424f [sys.catalog]: This master's current role is: LEADER
I20260927 12:44:55.780768 20700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1edafcb6b9124e98903bde58d8b3424f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1edafcb6b9124e98903bde58d8b3424f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1edafcb6b9124e98903bde58d8b3424f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46077 } } }
I20260927 12:44:55.780870 20700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1edafcb6b9124e98903bde58d8b3424f [sys.catalog]: This master's current role is: LEADER
I20260927 12:44:55.780951 20708 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:44:55.782157 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 12:44:55.784224 20708 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:44:55.790344 20708 catalog_manager.cc:1347] Generated new cluster ID: 9412d225e93644a68d5e1a473f819111
I20260927 12:44:55.790419 20708 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:44:55.793854 20697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:55.784568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44347 (local address 127.0.0.1:46077)
0927 12:44:55.784677 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:55.784681 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:55.784697 (+ 16us) server_negotiation.cc:407] Connection header received
0927 12:44:55.784793 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:55.784796 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:55.784842 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:55.784921 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:44:55.785848 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:55.786658 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:55.787762 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:55.788246 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:55.788369 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:55.790456 (+ 2087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:44:55.790460 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:44:55.790463 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:44:55.790490 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:44:55.792033 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:44:55.792518 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:44:55.792522 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:44:55.792524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:44:55.792609 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:44:55.792952 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:44:55.792955 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:44:55.792957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:44:55.793137 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:44:55.793290 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:55.793554 (+ 264us) server_negotiation.cc:299] Negotiation successful
0927 12:44:55.793682 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 12:44:55.811924 20708 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:44:55.812878 20708 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:44:55.819996 20708 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1edafcb6b9124e98903bde58d8b3424f: Generated new TSK 0
I20260927 12:44:55.820860 20708 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:44:55.830858 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 20626
W20260927 12:44:55.835718 20611 connection.cc:535] client connection to 127.0.0.1:46077 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46077): Connection reset by peer (error 104)
2026-09-27T12:44:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1414 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790512840836255-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:44:55 dist-test-slave-bnc1 krb5kdc[20725](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:44:55 dist-test-slave-bnc1 krb5kdc[20725](info): set up 2 sockets
Sep 27 12:44:55 dist-test-slave-bnc1 krb5kdc[20725](info): commencing operation
krb5kdc: starting...
W20260927 12:44:56.894325 392 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790512840836255-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790512840836255-392-0/krb5kdc/test-user.keytab.
Sep 27 12:44:56 dist-test-slave-bnc1 krb5kdc[20725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:44:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:44:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790512840836255-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790512840836255-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 12:44:57.020522 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790512840836255-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790512840836255-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790512840836255-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43053
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790512840836255-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43053
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:44:57.131111 20738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:44:57.131450 20738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:44:57.131544 20738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:44:57.135767 20738 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:44:57.135870 20738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:44:57.135911 20738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:44:57.135972 20738 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:44:57.136013 20738 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:44:57.141678 20738 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790512840836255-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790512840836255-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43053
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790512840836255-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20738
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790512840836255-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:44:57.142840 20738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:44:57.143950 20738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:44:57.153205 20743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:44:57.153322 20738 server_base.cc:1034] running on GCE node
W20260927 12:44:57.153205 20744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:44:57.153249 20746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:44:57.154213 20738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:44:57.155303 20738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:44:57.156466 20738 hybrid_clock.cc:648] HybridClock initialized: now 1790513097156455 us; error 35 us; skew 500 ppm
Sep 27 12:44:57 dist-test-slave-bnc1 krb5kdc[20725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513097, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:44:57.163522 20738 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 12:44:57.164532 20738 webserver.cc:460] Webserver started at http://127.0.0.1:45201/ using document root <none> and password file <none>
I20260927 12:44:57.165125 20738 fs_manager.cc:359] Metadata directory not provided
I20260927 12:44:57.165190 20738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:44:57.165437 20738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:44:57.167557 20738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790512840836255-392-0/minicluster-data/master-0/data/instance:
uuid: "2744af54cd574083902c259916af235d"
format_stamp: "Formatted at 2026-09-27 12:44:57 on dist-test-slave-bnc1"
server_key: "1b28b8f00db51c364f963d8e2aa3b441"
server_key_iv: "436085152257f0df5fc264016355cff6"
server_key_version: "encryptionkey@0"
I20260927 12:44:57.168126 20738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790512840836255-392-0/minicluster-data/master-0/wal/instance:
uuid: "2744af54cd574083902c259916af235d"
format_stamp: "Formatted at 2026-09-27 12:44:57 on dist-test-slave-bnc1"
server_key: "1b28b8f00db51c364f963d8e2aa3b441"
server_key_iv: "436085152257f0df5fc264016355cff6"
server_key_version: "encryptionkey@0"
I20260927 12:44:57.171828 20738 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260927 12:44:57.174238 20753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:44:57.175441 20738 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 12:44:57.175588 20738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790512840836255-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790512840836255-392-0/minicluster-data/master-0/wal
uuid: "2744af54cd574083902c259916af235d"
format_stamp: "Formatted at 2026-09-27 12:44:57 on dist-test-slave-bnc1"
server_key: "1b28b8f00db51c364f963d8e2aa3b441"
server_key_iv: "436085152257f0df5fc264016355cff6"
server_key_version: "encryptionkey@0"
I20260927 12:44:57.175699 20738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790512840836255-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790512840836255-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790512840836255-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:44:57.194511 20738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:44:57.197243 20738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:44:57.197443 20738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:44:57.204227 20738 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43053
I20260927 12:44:57.204252 20805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43053 every 8 connection(s)
I20260927 12:44:57.205192 20738 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:44:57.207026 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 20738
I20260927 12:44:57.207197 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790512840836255-392-0/minicluster-data/master-0/wal/instance
I20260927 12:44:57.207538 392 external_mini_cluster.cc:1428] Setting key 310292da279f361c65bc17a400899e6b
I20260927 12:44:57.208280 20806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:44:57.215034 20806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2744af54cd574083902c259916af235d: Bootstrap starting.
I20260927 12:44:57.217406 20809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:57.208981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49689 (local address 127.0.0.1:43053)
0927 12:44:57.209476 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:57.209493 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:57.209532 (+ 39us) server_negotiation.cc:407] Connection header received
0927 12:44:57.210277 (+ 745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:57.210294 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:57.210666 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:57.210971 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:44:57.211307 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:57.212414 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:57.213684 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:57.214222 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:57.214372 (+ 150us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:57.215389 (+ 1017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:44:57.215396 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:44:57.215412 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:44:57.215891 (+ 479us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:44:57.215960 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:44:57.215990 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:44:57.216135 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:57.216344 (+ 209us) server_negotiation.cc:299] Negotiation successful
0927 12:44:57.216486 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":152,"threads_started":1}
I20260927 12:44:57.217818 20806 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2744af54cd574083902c259916af235d: Neither blocks nor log segments found. Creating new log.
I20260927 12:44:57.218676 20806 log.cc:824] T 00000000000000000000000000000000 P 2744af54cd574083902c259916af235d: Log is configured to *not* fsync() on all Append() calls
I20260927 12:44:57.220598 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 12:44:57.220911 20806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2744af54cd574083902c259916af235d: No bootstrap required, opened a new log
I20260927 12:44:57.224467 20806 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2744af54cd574083902c259916af235d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2744af54cd574083902c259916af235d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43053 } }
I20260927 12:44:57.224710 20806 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2744af54cd574083902c259916af235d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:44:57.224792 20806 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2744af54cd574083902c259916af235d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2744af54cd574083902c259916af235d, State: Initialized, Role: FOLLOWER
I20260927 12:44:57.225203 20806 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2744af54cd574083902c259916af235d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2744af54cd574083902c259916af235d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43053 } }
I20260927 12:44:57.225347 20806 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2744af54cd574083902c259916af235d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:44:57.225430 20806 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2744af54cd574083902c259916af235d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:44:57.225530 20806 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2744af54cd574083902c259916af235d [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:44:57.226434 20806 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2744af54cd574083902c259916af235d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2744af54cd574083902c259916af235d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43053 } }
I20260927 12:44:57.226776 20806 leader_election.cc:302] T 00000000000000000000000000000000 P 2744af54cd574083902c259916af235d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2744af54cd574083902c259916af235d; no voters:
I20260927 12:44:57.227097 20806 leader_election.cc:288] T 00000000000000000000000000000000 P 2744af54cd574083902c259916af235d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:44:57.227176 20816 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2744af54cd574083902c259916af235d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:44:57.227384 20809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:57.222835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59985 (local address 127.0.0.1:43053)
0927 12:44:57.222968 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:57.222973 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:57.223201 (+ 228us) server_negotiation.cc:407] Connection header received
0927 12:44:57.223367 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:57.223370 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:57.223422 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:57.223509 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:44:57.223797 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:57.224364 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:57.225445 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:44:57.225811 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:44:57.225935 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:44:57.226872 (+ 937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:44:57.226876 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:44:57.226879 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:44:57.226894 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:44:57.226924 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:44:57.226937 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:44:57.227031 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:57.227176 (+ 145us) server_negotiation.cc:299] Negotiation successful
0927 12:44:57.227215 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 12:44:57.227416 20816 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2744af54cd574083902c259916af235d [term 1 LEADER]: Becoming Leader. State: Replica: 2744af54cd574083902c259916af235d, State: Running, Role: LEADER
I20260927 12:44:57.228031 20816 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2744af54cd574083902c259916af235d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2744af54cd574083902c259916af235d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43053 } }
I20260927 12:44:57.228479 20806 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2744af54cd574083902c259916af235d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:44:57.229875 20817 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2744af54cd574083902c259916af235d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2744af54cd574083902c259916af235d. Latest consensus state: current_term: 1 leader_uuid: "2744af54cd574083902c259916af235d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2744af54cd574083902c259916af235d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43053 } } }
I20260927 12:44:57.230001 20817 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2744af54cd574083902c259916af235d [sys.catalog]: This master's current role is: LEADER
I20260927 12:44:57.230293 20818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2744af54cd574083902c259916af235d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2744af54cd574083902c259916af235d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2744af54cd574083902c259916af235d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43053 } } }
I20260927 12:44:57.230381 20826 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:44:57.230464 20818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2744af54cd574083902c259916af235d [sys.catalog]: This master's current role is: LEADER
I20260927 12:44:57.233903 20826 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:44:57.239322 20826 catalog_manager.cc:1347] Generated new cluster ID: dbb11db2011e4dd492e80e94efe2db26
I20260927 12:44:57.239403 20826 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:44:57.244860 20826 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:44:57.246078 20826 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:44:57.255038 20826 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2744af54cd574083902c259916af235d: Generated new TSK 0
I20260927 12:44:57.255884 20826 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:44:57.264627 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 20738
W20260927 12:44:57.270092 20723 connection.cc:535] client connection to 127.0.0.1:43053 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43053): Connection reset by peer (error 104)
2026-09-27T12:44:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1434 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790512840836255-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:44:57 dist-test-slave-bnc1 krb5kdc[20837](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:44:57 dist-test-slave-bnc1 krb5kdc[20837](info): set up 2 sockets
Sep 27 12:44:57 dist-test-slave-bnc1 krb5kdc[20837](info): commencing operation
krb5kdc: starting...
W20260927 12:44:58.317834 392 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790512840836255-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790512840836255-392-0/krb5kdc/test-user.keytab.
Sep 27 12:44:58 dist-test-slave-bnc1 krb5kdc[20837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:44:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:44:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790512840836255-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790512840836255-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 12:44:58.438848 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790512840836255-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790512840836255-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790512840836255-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39665
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790512840836255-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39665
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:44:58.548224 20850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:44:58.548492 20850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:44:58.548535 20850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:44:58.552691 20850 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:44:58.552762 20850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:44:58.552781 20850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:44:58.552798 20850 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:44:58.552841 20850 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:44:58.558350 20850 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790512840836255-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790512840836255-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39665
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790512840836255-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39665
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790512840836255-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:44:58.559442 20850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:44:58.560503 20850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 12:44:58.569451 20850 server_base.cc:1034] running on GCE node
W20260927 12:44:58.569413 20856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:44:58.569420 20858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:44:58.569404 20855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:44:58.570400 20850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:44:58.571434 20850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:44:58.572597 20850 hybrid_clock.cc:648] HybridClock initialized: now 1790513098572581 us; error 32 us; skew 500 ppm
Sep 27 12:44:58 dist-test-slave-bnc1 krb5kdc[20837](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513098, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:44:58.575768 20850 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 12:44:58.576655 20850 webserver.cc:460] Webserver started at http://127.0.0.1:32831/ using document root <none> and password file <none>
I20260927 12:44:58.577219 20850 fs_manager.cc:359] Metadata directory not provided
I20260927 12:44:58.577309 20850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:44:58.577514 20850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:44:58.579574 20850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790512840836255-392-0/minicluster-data/master-0/data/instance:
uuid: "f63502dfe3274459946f9c807e4023af"
format_stamp: "Formatted at 2026-09-27 12:44:58 on dist-test-slave-bnc1"
server_key: "1d2c58680fb76572d80ec9e51c1bf26b"
server_key_iv: "dfdd9704eb3d45e49d4990057a24b394"
server_key_version: "encryptionkey@0"
I20260927 12:44:58.580139 20850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790512840836255-392-0/minicluster-data/master-0/wal/instance:
uuid: "f63502dfe3274459946f9c807e4023af"
format_stamp: "Formatted at 2026-09-27 12:44:58 on dist-test-slave-bnc1"
server_key: "1d2c58680fb76572d80ec9e51c1bf26b"
server_key_iv: "dfdd9704eb3d45e49d4990057a24b394"
server_key_version: "encryptionkey@0"
I20260927 12:44:58.584018 20850 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260927 12:44:58.586726 20865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:44:58.588070 20850 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 12:44:58.588248 20850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790512840836255-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790512840836255-392-0/minicluster-data/master-0/wal
uuid: "f63502dfe3274459946f9c807e4023af"
format_stamp: "Formatted at 2026-09-27 12:44:58 on dist-test-slave-bnc1"
server_key: "1d2c58680fb76572d80ec9e51c1bf26b"
server_key_iv: "dfdd9704eb3d45e49d4990057a24b394"
server_key_version: "encryptionkey@0"
I20260927 12:44:58.588364 20850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790512840836255-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790512840836255-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790512840836255-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:44:58.593057 20850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:44:58.596016 20850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:44:58.596220 20850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:44:58.603145 20850 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39665
I20260927 12:44:58.603222 20917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39665 every 8 connection(s)
I20260927 12:44:58.604087 20850 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:44:58.604768 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 20850
I20260927 12:44:58.604911 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790512840836255-392-0/minicluster-data/master-0/wal/instance
I20260927 12:44:58.605214 392 external_mini_cluster.cc:1428] Setting key 37067242259d4f58f224e3cf3631d841
I20260927 12:44:58.607519 20918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:44:58.610571 20921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:58.606767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46257 (local address 127.0.0.1:39665)
0927 12:44:58.607238 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:58.607261 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:58.607329 (+ 68us) server_negotiation.cc:407] Connection header received
0927 12:44:58.608076 (+ 747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:58.608093 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:58.608370 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:58.608721 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:44:58.609401 (+ 680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:44:58.609406 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:44:58.609421 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:44:58.609431 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:44:58.609492 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:44:58.609510 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:44:58.609687 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:58.609770 (+ 83us) server_negotiation.cc:299] Negotiation successful
0927 12:44:58.609914 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":133,"threads_started":1}
I20260927 12:44:58.613372 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 12:44:58.615679 20918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f63502dfe3274459946f9c807e4023af: Bootstrap starting.
I20260927 12:44:58.617906 20921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:44:58.615684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56673 (local address 127.0.0.1:39665)
0927 12:44:58.615805 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0927 12:44:58.615809 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:44:58.616088 (+ 279us) server_negotiation.cc:407] Connection header received
0927 12:44:58.616213 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:44:58.616216 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:44:58.616267 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:44:58.616336 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:44:58.617421 (+ 1085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:44:58.617424 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:44:58.617427 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:44:58.617438 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:44:58.617463 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:44:58.617480 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:44:58.617606 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 12:44:58.617735 (+ 129us) server_negotiation.cc:299] Negotiation successful
0927 12:44:58.617774 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 12:44:58.618517 20918 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f63502dfe3274459946f9c807e4023af: Neither blocks nor log segments found. Creating new log.
I20260927 12:44:58.619400 20918 log.cc:824] T 00000000000000000000000000000000 P f63502dfe3274459946f9c807e4023af: Log is configured to *not* fsync() on all Append() calls
I20260927 12:44:58.621498 20918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f63502dfe3274459946f9c807e4023af: No bootstrap required, opened a new log
I20260927 12:44:58.625051 20918 raft_consensus.cc:348] T 00000000000000000000000000000000 P f63502dfe3274459946f9c807e4023af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f63502dfe3274459946f9c807e4023af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39665 } }
I20260927 12:44:58.625373 20918 raft_consensus.cc:374] T 00000000000000000000000000000000 P f63502dfe3274459946f9c807e4023af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:44:58.625454 20918 raft_consensus.cc:729] T 00000000000000000000000000000000 P f63502dfe3274459946f9c807e4023af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f63502dfe3274459946f9c807e4023af, State: Initialized, Role: FOLLOWER
I20260927 12:44:58.625963 20918 consensus_queue.cc:260] T 00000000000000000000000000000000 P f63502dfe3274459946f9c807e4023af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f63502dfe3274459946f9c807e4023af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39665 } }
I20260927 12:44:58.626096 20918 raft_consensus.cc:388] T 00000000000000000000000000000000 P f63502dfe3274459946f9c807e4023af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:44:58.626140 20918 raft_consensus.cc:482] T 00000000000000000000000000000000 P f63502dfe3274459946f9c807e4023af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:44:58.626209 20918 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f63502dfe3274459946f9c807e4023af [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:44:58.627243 20918 raft_consensus.cc:504] T 00000000000000000000000000000000 P f63502dfe3274459946f9c807e4023af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f63502dfe3274459946f9c807e4023af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39665 } }
I20260927 12:44:58.627578 20918 leader_election.cc:302] T 00000000000000000000000000000000 P f63502dfe3274459946f9c807e4023af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f63502dfe3274459946f9c807e4023af; no voters:
I20260927 12:44:58.627902 20918 leader_election.cc:288] T 00000000000000000000000000000000 P f63502dfe3274459946f9c807e4023af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:44:58.628041 20928 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f63502dfe3274459946f9c807e4023af [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:44:58.628319 20928 raft_consensus.cc:686] T 00000000000000000000000000000000 P f63502dfe3274459946f9c807e4023af [term 1 LEADER]: Becoming Leader. State: Replica: f63502dfe3274459946f9c807e4023af, State: Running, Role: LEADER
I20260927 12:44:58.628687 20928 consensus_queue.cc:237] T 00000000000000000000000000000000 P f63502dfe3274459946f9c807e4023af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f63502dfe3274459946f9c807e4023af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39665 } }
I20260927 12:44:58.629288 20918 sys_catalog.cc:564] T 00000000000000000000000000000000 P f63502dfe3274459946f9c807e4023af [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:44:58.630630 20930 sys_catalog.cc:455] T 00000000000000000000000000000000 P f63502dfe3274459946f9c807e4023af [sys.catalog]: SysCatalogTable state changed. Reason: New leader f63502dfe3274459946f9c807e4023af. Latest consensus state: current_term: 1 leader_uuid: "f63502dfe3274459946f9c807e4023af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f63502dfe3274459946f9c807e4023af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39665 } } }
I20260927 12:44:58.630782 20930 sys_catalog.cc:458] T 00000000000000000000000000000000 P f63502dfe3274459946f9c807e4023af [sys.catalog]: This master's current role is: LEADER
I20260927 12:44:58.631032 20929 sys_catalog.cc:455] T 00000000000000000000000000000000 P f63502dfe3274459946f9c807e4023af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f63502dfe3274459946f9c807e4023af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f63502dfe3274459946f9c807e4023af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39665 } } }
I20260927 12:44:58.631124 20929 sys_catalog.cc:458] T 00000000000000000000000000000000 P f63502dfe3274459946f9c807e4023af [sys.catalog]: This master's current role is: LEADER
I20260927 12:44:58.631268 20937 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:44:58.634100 20937 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:44:58.639339 20937 catalog_manager.cc:1347] Generated new cluster ID: a77ad078cf5f4d8bb032616e792144e6
I20260927 12:44:58.639415 20937 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:44:58.654861 20937 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:44:58.655210 20937 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:44:58.660383 20937 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f63502dfe3274459946f9c807e4023af: Generated new TSK 0
I20260927 12:44:58.661158 20937 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:44:58.673372 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 20850
2026-09-27T12:44:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1408 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790512840836255-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:44:58 dist-test-slave-bnc1 krb5kdc[20949](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:44:58 dist-test-slave-bnc1 krb5kdc[20949](info): set up 2 sockets
Sep 27 12:44:58 dist-test-slave-bnc1 krb5kdc[20949](info): commencing operation
krb5kdc: starting...
W20260927 12:44:59.713872 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790512840836255-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790512840836255-392-0/krb5kdc/test-user.keytab.
Sep 27 12:44:59 dist-test-slave-bnc1 krb5kdc[20949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513099, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:44:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:44:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790512840836255-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790512840836255-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 12:44:59.838394 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790512840836255-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790512840836255-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790512840836255-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46379
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790512840836255-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46379
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:44:59.951126 20962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:44:59.951388 20962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:44:59.951431 20962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:44:59.956013 20962 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:44:59.956089 20962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:44:59.956111 20962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:44:59.956128 20962 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:44:59.956187 20962 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:44:59.961740 20962 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790512840836255-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790512840836255-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46379
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790512840836255-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790512840836255-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:44:59.963299 20962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:44:59.964365 20962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:44:59.973244 20967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:44:59.973244 20970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:44:59.973254 20968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:44:59.973500 20962 server_base.cc:1034] running on GCE node
I20260927 12:44:59.974169 20962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:44:59.975319 20962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:44:59.976472 20962 hybrid_clock.cc:648] HybridClock initialized: now 1790513099976441 us; error 45 us; skew 500 ppm
Sep 27 12:44:59 dist-test-slave-bnc1 krb5kdc[20949](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513099, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:44:59.979288 20962 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 12:44:59.980227 20962 webserver.cc:460] Webserver started at http://127.0.0.1:45805/ using document root <none> and password file <none>
I20260927 12:44:59.980798 20962 fs_manager.cc:359] Metadata directory not provided
I20260927 12:44:59.980890 20962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:44:59.981122 20962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:44:59.983201 20962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790512840836255-392-0/minicluster-data/master-0/data/instance:
uuid: "66aea241842e4dd3a855e5788df2a54c"
format_stamp: "Formatted at 2026-09-27 12:44:59 on dist-test-slave-bnc1"
server_key: "94eb42ee3bb08e9eab7740d7058b56f9"
server_key_iv: "f0009e9c1127507142ca88405a0c94de"
server_key_version: "encryptionkey@0"
I20260927 12:44:59.983768 20962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790512840836255-392-0/minicluster-data/master-0/wal/instance:
uuid: "66aea241842e4dd3a855e5788df2a54c"
format_stamp: "Formatted at 2026-09-27 12:44:59 on dist-test-slave-bnc1"
server_key: "94eb42ee3bb08e9eab7740d7058b56f9"
server_key_iv: "f0009e9c1127507142ca88405a0c94de"
server_key_version: "encryptionkey@0"
I20260927 12:44:59.987443 20962 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20260927 12:44:59.989734 20977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:44:59.990929 20962 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:44:59.991076 20962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790512840836255-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790512840836255-392-0/minicluster-data/master-0/wal
uuid: "66aea241842e4dd3a855e5788df2a54c"
format_stamp: "Formatted at 2026-09-27 12:44:59 on dist-test-slave-bnc1"
server_key: "94eb42ee3bb08e9eab7740d7058b56f9"
server_key_iv: "f0009e9c1127507142ca88405a0c94de"
server_key_version: "encryptionkey@0"
I20260927 12:44:59.991197 20962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790512840836255-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790512840836255-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790512840836255-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:45:00.010562 20962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:45:00.013644 20962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:45:00.013856 20962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:45:00.021230 20962 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46379
I20260927 12:45:00.021246 21029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46379 every 8 connection(s)
I20260927 12:45:00.022274 20962 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:45:00.024989 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 20962
I20260927 12:45:00.025146 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790512840836255-392-0/minicluster-data/master-0/wal/instance
I20260927 12:45:00.025199 21030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:45:00.025461 392 external_mini_cluster.cc:1428] Setting key bec168c4119aa4b4815d6afd2fa17cd3
I20260927 12:45:00.032217 21030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66aea241842e4dd3a855e5788df2a54c: Bootstrap starting.
Sep 27 12:45:00 dist-test-slave-bnc1 krb5kdc[20949](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513099, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 12:45:00.035068 21030 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66aea241842e4dd3a855e5788df2a54c: Neither blocks nor log segments found. Creating new log.
I20260927 12:45:00.035995 21030 log.cc:824] T 00000000000000000000000000000000 P 66aea241842e4dd3a855e5788df2a54c: Log is configured to *not* fsync() on all Append() calls
I20260927 12:45:00.038331 21030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66aea241842e4dd3a855e5788df2a54c: No bootstrap required, opened a new log
I20260927 12:45:00.040603 21033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:00.026957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59099 (local address 127.0.0.1:46379)
0927 12:45:00.027383 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:00.027397 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:00.027431 (+ 34us) server_negotiation.cc:407] Connection header received
0927 12:45:00.028049 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:00.028105 (+ 56us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:00.028441 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:00.028729 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:00.029690 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:00.030659 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:00.031827 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:00.032269 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:00.032449 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:45:00.035307 (+ 2858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:45:00.035312 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:45:00.035326 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:45:00.035353 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:45:00.037420 (+ 2067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:45:00.037911 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:45:00.037917 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:45:00.037923 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:45:00.038063 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:45:00.038335 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:45:00.038340 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:45:00.038345 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:45:00.038855 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:45:00.039045 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0927 12:45:00.039314 (+ 269us) server_negotiation.cc:299] Negotiation successful
0927 12:45:00.039610 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":138,"threads_started":1}
I20260927 12:45:00.042831 21030 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66aea241842e4dd3a855e5788df2a54c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66aea241842e4dd3a855e5788df2a54c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46379 } }
I20260927 12:45:00.043119 21030 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66aea241842e4dd3a855e5788df2a54c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:45:00.043195 21030 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66aea241842e4dd3a855e5788df2a54c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66aea241842e4dd3a855e5788df2a54c, State: Initialized, Role: FOLLOWER
I20260927 12:45:00.043644 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 12:45:00.043686 21030 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66aea241842e4dd3a855e5788df2a54c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66aea241842e4dd3a855e5788df2a54c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46379 } }
I20260927 12:45:00.043833 21030 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66aea241842e4dd3a855e5788df2a54c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:45:00.043889 21030 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66aea241842e4dd3a855e5788df2a54c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:45:00.043979 21030 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66aea241842e4dd3a855e5788df2a54c [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:45:00.045253 21030 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66aea241842e4dd3a855e5788df2a54c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66aea241842e4dd3a855e5788df2a54c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46379 } }
I20260927 12:45:00.045661 21030 leader_election.cc:302] T 00000000000000000000000000000000 P 66aea241842e4dd3a855e5788df2a54c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66aea241842e4dd3a855e5788df2a54c; no voters:
I20260927 12:45:00.046036 21030 leader_election.cc:288] T 00000000000000000000000000000000 P 66aea241842e4dd3a855e5788df2a54c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:45:00.046228 21039 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66aea241842e4dd3a855e5788df2a54c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:45:00.046525 21039 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66aea241842e4dd3a855e5788df2a54c [term 1 LEADER]: Becoming Leader. State: Replica: 66aea241842e4dd3a855e5788df2a54c, State: Running, Role: LEADER
I20260927 12:45:00.046818 21039 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66aea241842e4dd3a855e5788df2a54c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66aea241842e4dd3a855e5788df2a54c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46379 } }
I20260927 12:45:00.047384 21030 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66aea241842e4dd3a855e5788df2a54c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:45:00.048480 21041 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66aea241842e4dd3a855e5788df2a54c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66aea241842e4dd3a855e5788df2a54c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66aea241842e4dd3a855e5788df2a54c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46379 } } }
I20260927 12:45:00.048671 21041 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66aea241842e4dd3a855e5788df2a54c [sys.catalog]: This master's current role is: LEADER
I20260927 12:45:00.049026 21042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66aea241842e4dd3a855e5788df2a54c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66aea241842e4dd3a855e5788df2a54c. Latest consensus state: current_term: 1 leader_uuid: "66aea241842e4dd3a855e5788df2a54c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66aea241842e4dd3a855e5788df2a54c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46379 } } }
I20260927 12:45:00.050022 21042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66aea241842e4dd3a855e5788df2a54c [sys.catalog]: This master's current role is: LEADER
W20260927 12:45:00.052595 21055 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 66aea241842e4dd3a855e5788df2a54c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 12:45:00.052692 21055 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 12:45:00.052781 21049 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:45:00.053848 21049 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:45:00.056674 21033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:00.045984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59543 (local address 127.0.0.1:46379)
0927 12:45:00.046137 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:00.046142 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:00.048142 (+ 2000us) server_negotiation.cc:407] Connection header received
0927 12:45:00.048352 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:00.048355 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:00.048403 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:00.048459 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:00.049459 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:00.050181 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:00.051028 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:00.051363 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:00.051471 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:45:00.053073 (+ 1602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:45:00.053077 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:45:00.053080 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:45:00.053104 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:45:00.054855 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:45:00.055349 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:45:00.055352 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:45:00.055354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:45:00.055442 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:45:00.055748 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:45:00.055752 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:45:00.055755 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:45:00.056022 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:45:00.056149 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 12:45:00.056424 (+ 275us) server_negotiation.cc:299] Negotiation successful
0927 12:45:00.056521 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 12:45:00.060518 21049 catalog_manager.cc:1347] Generated new cluster ID: 7c1bdebb26724c41a3e7e73efa5a72a8
I20260927 12:45:00.060592 21049 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:45:00.072835 21049 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:45:00.073997 21049 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:45:00.082012 21049 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 66aea241842e4dd3a855e5788df2a54c: Generated new TSK 0
I20260927 12:45:00.082741 21049 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:45:00.087041 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 20962
2026-09-27T12:45:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1413 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790512840836255-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:45:00 dist-test-slave-bnc1 krb5kdc[21061](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:45:00 dist-test-slave-bnc1 krb5kdc[21061](info): set up 2 sockets
Sep 27 12:45:00 dist-test-slave-bnc1 krb5kdc[21061](info): commencing operation
krb5kdc: starting...
W20260927 12:45:01.117969 392 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790512840836255-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790512840836255-392-0/krb5kdc/test-user.keytab.
Sep 27 12:45:01 dist-test-slave-bnc1 krb5kdc[21061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:45:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:45:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790512840836255-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790512840836255-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 12:45:01.238119 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790512840836255-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790512840836255-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790512840836255-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42393
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790512840836255-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42393
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:45:01.348937 21074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:45:01.349254 21074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:45:01.349336 21074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:45:01.353307 21074 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:45:01.353403 21074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:45:01.353444 21074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:45:01.353504 21074 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:45:01.353543 21074 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:45:01.358930 21074 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790512840836255-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790512840836255-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42393
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790512840836255-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21074
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790512840836255-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:45:01.360054 21074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:45:01.361136 21074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 12:45:01.369796 21074 server_base.cc:1034] running on GCE node
W20260927 12:45:01.369753 21080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:45:01.369745 21082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:45:01.369741 21079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:45:01.370576 21074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:45:01.371709 21074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:45:01.372936 21074 hybrid_clock.cc:648] HybridClock initialized: now 1790513101372909 us; error 44 us; skew 500 ppm
Sep 27 12:45:01 dist-test-slave-bnc1 krb5kdc[21061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513101, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:45:01.375912 21074 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 12:45:01.376883 21074 webserver.cc:460] Webserver started at http://127.0.0.1:36977/ using document root <none> and password file <none>
I20260927 12:45:01.377453 21074 fs_manager.cc:359] Metadata directory not provided
I20260927 12:45:01.377545 21074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:45:01.377811 21074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:45:01.379871 21074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790512840836255-392-0/minicluster-data/master-0/data/instance:
uuid: "a92f51d5944141a19790d26a31f8d66b"
format_stamp: "Formatted at 2026-09-27 12:45:01 on dist-test-slave-bnc1"
server_key: "6506134f403c2a79469ffcce6a18cadc"
server_key_iv: "7f8f577ac9b8cac2f318e9e055125b32"
server_key_version: "encryptionkey@0"
I20260927 12:45:01.380446 21074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790512840836255-392-0/minicluster-data/master-0/wal/instance:
uuid: "a92f51d5944141a19790d26a31f8d66b"
format_stamp: "Formatted at 2026-09-27 12:45:01 on dist-test-slave-bnc1"
server_key: "6506134f403c2a79469ffcce6a18cadc"
server_key_iv: "7f8f577ac9b8cac2f318e9e055125b32"
server_key_version: "encryptionkey@0"
I20260927 12:45:01.384277 21074 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 12:45:01.386901 21089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:45:01.388244 21074 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 12:45:01.388399 21074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790512840836255-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790512840836255-392-0/minicluster-data/master-0/wal
uuid: "a92f51d5944141a19790d26a31f8d66b"
format_stamp: "Formatted at 2026-09-27 12:45:01 on dist-test-slave-bnc1"
server_key: "6506134f403c2a79469ffcce6a18cadc"
server_key_iv: "7f8f577ac9b8cac2f318e9e055125b32"
server_key_version: "encryptionkey@0"
I20260927 12:45:01.388525 21074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790512840836255-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790512840836255-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790512840836255-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:45:01.405409 21074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:45:01.408501 21074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:45:01.408726 21074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:45:01.416306 21074 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42393
I20260927 12:45:01.416332 21141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42393 every 8 connection(s)
I20260927 12:45:01.417305 21074 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:45:01.420248 21142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:45:01.424280 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 21074
I20260927 12:45:01.424423 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790512840836255-392-0/minicluster-data/master-0/wal/instance
I20260927 12:45:01.424719 392 external_mini_cluster.cc:1428] Setting key 4f2c39656a1600536cb5d6e44032e0f6
I20260927 12:45:01.426170 21142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a92f51d5944141a19790d26a31f8d66b: Bootstrap starting.
I20260927 12:45:01.428514 21142 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a92f51d5944141a19790d26a31f8d66b: Neither blocks nor log segments found. Creating new log.
I20260927 12:45:01.429275 21142 log.cc:824] T 00000000000000000000000000000000 P a92f51d5944141a19790d26a31f8d66b: Log is configured to *not* fsync() on all Append() calls
I20260927 12:45:01.431494 21142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a92f51d5944141a19790d26a31f8d66b: No bootstrap required, opened a new log
I20260927 12:45:01.434468 21145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:01.426130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46367 (local address 127.0.0.1:42393)
0927 12:45:01.426608 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:01.426624 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:01.426660 (+ 36us) server_negotiation.cc:407] Connection header received
0927 12:45:01.427292 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:01.427309 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:01.427682 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:01.427971 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:01.428298 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:01.429349 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:01.430571 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:01.431058 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:01.431188 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:45:01.432150 (+ 962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:45:01.432155 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:45:01.432168 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:45:01.432561 (+ 393us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:45:01.432618 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:45:01.432641 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:45:01.432719 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 12:45:01.432886 (+ 167us) server_negotiation.cc:299] Negotiation successful
0927 12:45:01.433268 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":155,"threads_started":1}
I20260927 12:45:01.434959 21142 raft_consensus.cc:348] T 00000000000000000000000000000000 P a92f51d5944141a19790d26a31f8d66b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a92f51d5944141a19790d26a31f8d66b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42393 } }
I20260927 12:45:01.435228 21142 raft_consensus.cc:374] T 00000000000000000000000000000000 P a92f51d5944141a19790d26a31f8d66b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:45:01.435329 21142 raft_consensus.cc:729] T 00000000000000000000000000000000 P a92f51d5944141a19790d26a31f8d66b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a92f51d5944141a19790d26a31f8d66b, State: Initialized, Role: FOLLOWER
I20260927 12:45:01.435778 21142 consensus_queue.cc:260] T 00000000000000000000000000000000 P a92f51d5944141a19790d26a31f8d66b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a92f51d5944141a19790d26a31f8d66b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42393 } }
I20260927 12:45:01.435904 21142 raft_consensus.cc:388] T 00000000000000000000000000000000 P a92f51d5944141a19790d26a31f8d66b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:45:01.436010 21142 raft_consensus.cc:482] T 00000000000000000000000000000000 P a92f51d5944141a19790d26a31f8d66b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:45:01.436133 21142 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a92f51d5944141a19790d26a31f8d66b [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:45:01.437224 21142 raft_consensus.cc:504] T 00000000000000000000000000000000 P a92f51d5944141a19790d26a31f8d66b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a92f51d5944141a19790d26a31f8d66b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42393 } }
I20260927 12:45:01.437633 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 12:45:01.437665 21142 leader_election.cc:302] T 00000000000000000000000000000000 P a92f51d5944141a19790d26a31f8d66b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a92f51d5944141a19790d26a31f8d66b; no voters:
I20260927 12:45:01.437997 21142 leader_election.cc:288] T 00000000000000000000000000000000 P a92f51d5944141a19790d26a31f8d66b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:45:01.438124 21147 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a92f51d5944141a19790d26a31f8d66b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:45:01.438360 21147 raft_consensus.cc:686] T 00000000000000000000000000000000 P a92f51d5944141a19790d26a31f8d66b [term 1 LEADER]: Becoming Leader. State: Replica: a92f51d5944141a19790d26a31f8d66b, State: Running, Role: LEADER
I20260927 12:45:01.438752 21147 consensus_queue.cc:237] T 00000000000000000000000000000000 P a92f51d5944141a19790d26a31f8d66b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a92f51d5944141a19790d26a31f8d66b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42393 } }
I20260927 12:45:01.439388 21142 sys_catalog.cc:564] T 00000000000000000000000000000000 P a92f51d5944141a19790d26a31f8d66b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:45:01.440471 21148 sys_catalog.cc:455] T 00000000000000000000000000000000 P a92f51d5944141a19790d26a31f8d66b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a92f51d5944141a19790d26a31f8d66b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a92f51d5944141a19790d26a31f8d66b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42393 } } }
I20260927 12:45:01.440609 21148 sys_catalog.cc:458] T 00000000000000000000000000000000 P a92f51d5944141a19790d26a31f8d66b [sys.catalog]: This master's current role is: LEADER
I20260927 12:45:01.440862 21148 sys_catalog.cc:455] T 00000000000000000000000000000000 P a92f51d5944141a19790d26a31f8d66b [sys.catalog]: SysCatalogTable state changed. Reason: New leader a92f51d5944141a19790d26a31f8d66b. Latest consensus state: current_term: 1 leader_uuid: "a92f51d5944141a19790d26a31f8d66b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a92f51d5944141a19790d26a31f8d66b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42393 } } }
I20260927 12:45:01.440948 21148 sys_catalog.cc:458] T 00000000000000000000000000000000 P a92f51d5944141a19790d26a31f8d66b [sys.catalog]: This master's current role is: LEADER
I20260927 12:45:01.441015 21161 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:45:01.444387 21161 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:45:01.445151 21145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:01.439906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59185 (local address 127.0.0.1:42393)
0927 12:45:01.440007 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:01.440014 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:01.440290 (+ 276us) server_negotiation.cc:407] Connection header received
0927 12:45:01.440487 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:01.440493 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:01.440546 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:01.440613 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:01.441095 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:01.441848 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:01.442981 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:01.443301 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:01.443401 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:45:01.444407 (+ 1006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:45:01.444411 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:45:01.444413 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:45:01.444425 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:45:01.444456 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:45:01.444470 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:45:01.444547 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 12:45:01.444986 (+ 439us) server_negotiation.cc:299] Negotiation successful
0927 12:45:01.445041 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260927 12:45:01.449970 21161 catalog_manager.cc:1347] Generated new cluster ID: 37ea3b70df354a0bba4c6a3d98b762b3
I20260927 12:45:01.450038 21161 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:45:01.476480 21161 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:45:01.477801 21161 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:45:01.485373 21161 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a92f51d5944141a19790d26a31f8d66b: Generated new TSK 0
I20260927 12:45:01.486239 21161 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:45:01.494011 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 21074
2026-09-27T12:45:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1406 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790512840836255-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:45:01 dist-test-slave-bnc1 krb5kdc[21173](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:45:01 dist-test-slave-bnc1 krb5kdc[21173](info): set up 2 sockets
Sep 27 12:45:01 dist-test-slave-bnc1 krb5kdc[21173](info): commencing operation
krb5kdc: starting...
W20260927 12:45:02.541864 392 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790512840836255-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790512840836255-392-0/krb5kdc/test-user.keytab.
Sep 27 12:45:02 dist-test-slave-bnc1 krb5kdc[21173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:45:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:45:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790512840836255-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790512840836255-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 12:45:02.668644 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790512840836255-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790512840836255-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790512840836255-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44971
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790512840836255-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44971
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:45:02.780890 21186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:45:02.781232 21186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:45:02.781342 21186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:45:02.785465 21186 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:45:02.785586 21186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:45:02.785629 21186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:45:02.785666 21186 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:45:02.785701 21186 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:45:02.791172 21186 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790512840836255-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790512840836255-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44971
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790512840836255-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21186
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790512840836255-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:45:02.792327 21186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:45:02.793370 21186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:45:02.802205 21194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:45:02.802205 21192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:45:02.802223 21191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:45:02.802637 21186 server_base.cc:1034] running on GCE node
I20260927 12:45:02.803277 21186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:45:02.804514 21186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:45:02.805684 21186 hybrid_clock.cc:648] HybridClock initialized: now 1790513102805672 us; error 26 us; skew 500 ppm
Sep 27 12:45:02 dist-test-slave-bnc1 krb5kdc[21173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513102, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:45:02.812871 21186 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 12:45:02.814010 21186 webserver.cc:460] Webserver started at http://127.0.0.1:42057/ using document root <none> and password file <none>
I20260927 12:45:02.814575 21186 fs_manager.cc:359] Metadata directory not provided
I20260927 12:45:02.814674 21186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:45:02.814909 21186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:45:02.816926 21186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790512840836255-392-0/minicluster-data/master-0/data/instance:
uuid: "a0acb9720a7c481f8ec8bcd7a0529255"
format_stamp: "Formatted at 2026-09-27 12:45:02 on dist-test-slave-bnc1"
server_key: "158b75f652c37ff06d419b32616b93fc"
server_key_iv: "7f1309dd66c90362c1aabb5f277de3da"
server_key_version: "encryptionkey@0"
I20260927 12:45:02.817495 21186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790512840836255-392-0/minicluster-data/master-0/wal/instance:
uuid: "a0acb9720a7c481f8ec8bcd7a0529255"
format_stamp: "Formatted at 2026-09-27 12:45:02 on dist-test-slave-bnc1"
server_key: "158b75f652c37ff06d419b32616b93fc"
server_key_iv: "7f1309dd66c90362c1aabb5f277de3da"
server_key_version: "encryptionkey@0"
I20260927 12:45:02.821329 21186 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 12:45:02.823865 21201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:45:02.825173 21186 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:45:02.825318 21186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790512840836255-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790512840836255-392-0/minicluster-data/master-0/wal
uuid: "a0acb9720a7c481f8ec8bcd7a0529255"
format_stamp: "Formatted at 2026-09-27 12:45:02 on dist-test-slave-bnc1"
server_key: "158b75f652c37ff06d419b32616b93fc"
server_key_iv: "7f1309dd66c90362c1aabb5f277de3da"
server_key_version: "encryptionkey@0"
I20260927 12:45:02.825438 21186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790512840836255-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790512840836255-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790512840836255-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:45:02.830255 21186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:45:02.833429 21186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:45:02.833686 21186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:45:02.841619 21186 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44971
I20260927 12:45:02.841650 21253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44971 every 8 connection(s)
I20260927 12:45:02.842650 21186 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:45:02.844621 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 21186
I20260927 12:45:02.844767 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790512840836255-392-0/minicluster-data/master-0/wal/instance
I20260927 12:45:02.845068 392 external_mini_cluster.cc:1428] Setting key 3fa15fdc78e955da476bb1184b41b9d6
I20260927 12:45:02.845839 21254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:45:02.850494 21257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:02.846558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46969 (local address 127.0.0.1:44971)
0927 12:45:02.847016 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:02.847032 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:02.847068 (+ 36us) server_negotiation.cc:407] Connection header received
0927 12:45:02.847791 (+ 723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:02.847808 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:02.848101 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:02.848446 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:02.849274 (+ 828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:45:02.849278 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:45:02.849290 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:45:02.849301 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:45:02.849358 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:45:02.849379 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:45:02.849543 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 12:45:02.849642 (+ 99us) server_negotiation.cc:299] Negotiation successful
0927 12:45:02.849799 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":134,"threads_started":1}
I20260927 12:45:02.852011 21254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0acb9720a7c481f8ec8bcd7a0529255: Bootstrap starting.
I20260927 12:45:02.853193 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 12:45:02.854405 21254 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0acb9720a7c481f8ec8bcd7a0529255: Neither blocks nor log segments found. Creating new log.
I20260927 12:45:02.855162 21254 log.cc:824] T 00000000000000000000000000000000 P a0acb9720a7c481f8ec8bcd7a0529255: Log is configured to *not* fsync() on all Append() calls
I20260927 12:45:02.857008 21257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:02.855303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43085 (local address 127.0.0.1:44971)
0927 12:45:02.855421 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:02.855426 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:02.855664 (+ 238us) server_negotiation.cc:407] Connection header received
0927 12:45:02.855789 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:02.855791 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:02.855818 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:02.855882 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:02.856660 (+ 778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:45:02.856663 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:45:02.856666 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:45:02.856677 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:45:02.856706 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:45:02.856722 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:45:02.856809 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 12:45:02.856871 (+ 62us) server_negotiation.cc:299] Negotiation successful
0927 12:45:02.856906 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 12:45:02.857409 21254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0acb9720a7c481f8ec8bcd7a0529255: No bootstrap required, opened a new log
I20260927 12:45:02.860846 21254 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0acb9720a7c481f8ec8bcd7a0529255 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0acb9720a7c481f8ec8bcd7a0529255" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44971 } }
I20260927 12:45:02.861086 21254 raft_consensus.cc:374] T 00000000000000000000000000000000 P a0acb9720a7c481f8ec8bcd7a0529255 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:45:02.861188 21254 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0acb9720a7c481f8ec8bcd7a0529255 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0acb9720a7c481f8ec8bcd7a0529255, State: Initialized, Role: FOLLOWER
I20260927 12:45:02.861640 21254 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0acb9720a7c481f8ec8bcd7a0529255 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0acb9720a7c481f8ec8bcd7a0529255" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44971 } }
I20260927 12:45:02.861804 21254 raft_consensus.cc:388] T 00000000000000000000000000000000 P a0acb9720a7c481f8ec8bcd7a0529255 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:45:02.861891 21254 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0acb9720a7c481f8ec8bcd7a0529255 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:45:02.861989 21254 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0acb9720a7c481f8ec8bcd7a0529255 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:45:02.862864 21254 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0acb9720a7c481f8ec8bcd7a0529255 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0acb9720a7c481f8ec8bcd7a0529255" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44971 } }
I20260927 12:45:02.863206 21254 leader_election.cc:302] T 00000000000000000000000000000000 P a0acb9720a7c481f8ec8bcd7a0529255 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a0acb9720a7c481f8ec8bcd7a0529255; no voters:
I20260927 12:45:02.863533 21254 leader_election.cc:288] T 00000000000000000000000000000000 P a0acb9720a7c481f8ec8bcd7a0529255 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:45:02.863636 21264 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0acb9720a7c481f8ec8bcd7a0529255 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:45:02.863804 21264 raft_consensus.cc:686] T 00000000000000000000000000000000 P a0acb9720a7c481f8ec8bcd7a0529255 [term 1 LEADER]: Becoming Leader. State: Replica: a0acb9720a7c481f8ec8bcd7a0529255, State: Running, Role: LEADER
I20260927 12:45:02.864147 21264 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0acb9720a7c481f8ec8bcd7a0529255 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0acb9720a7c481f8ec8bcd7a0529255" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44971 } }
I20260927 12:45:02.864827 21254 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0acb9720a7c481f8ec8bcd7a0529255 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:45:02.865869 21266 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0acb9720a7c481f8ec8bcd7a0529255 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0acb9720a7c481f8ec8bcd7a0529255. Latest consensus state: current_term: 1 leader_uuid: "a0acb9720a7c481f8ec8bcd7a0529255" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0acb9720a7c481f8ec8bcd7a0529255" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44971 } } }
I20260927 12:45:02.866011 21266 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0acb9720a7c481f8ec8bcd7a0529255 [sys.catalog]: This master's current role is: LEADER
I20260927 12:45:02.866307 21265 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0acb9720a7c481f8ec8bcd7a0529255 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a0acb9720a7c481f8ec8bcd7a0529255" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0acb9720a7c481f8ec8bcd7a0529255" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44971 } } }
I20260927 12:45:02.866384 21265 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0acb9720a7c481f8ec8bcd7a0529255 [sys.catalog]: This master's current role is: LEADER
I20260927 12:45:02.866415 21273 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:45:02.869656 21273 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:45:02.875530 21273 catalog_manager.cc:1347] Generated new cluster ID: 310cb03500484c16bf0977b5e3f9a1d4
I20260927 12:45:02.875618 21273 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:45:02.904489 21273 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:45:02.904886 21273 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:45:02.918110 21273 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a0acb9720a7c481f8ec8bcd7a0529255: Generated new TSK 0
I20260927 12:45:02.919049 21273 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:45:02.926491 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 21186
2026-09-27T12:45:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1432 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790512840836255-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:45:02 dist-test-slave-bnc1 krb5kdc[21285](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:45:02 dist-test-slave-bnc1 krb5kdc[21285](info): set up 2 sockets
Sep 27 12:45:02 dist-test-slave-bnc1 krb5kdc[21285](info): commencing operation
krb5kdc: starting...
W20260927 12:45:03.977960 392 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790512840836255-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790512840836255-392-0/krb5kdc/test-user.keytab.
Sep 27 12:45:04 dist-test-slave-bnc1 krb5kdc[21285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513104, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:45:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:45:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790512840836255-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790512840836255-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 12:45:04.101895 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790512840836255-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790512840836255-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790512840836255-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45133
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790512840836255-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45133
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:45:04.216240 21298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:45:04.216537 21298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:45:04.216583 21298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:45:04.220801 21298 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:45:04.220875 21298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:45:04.220897 21298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:45:04.220917 21298 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:45:04.220993 21298 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:45:04.226449 21298 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790512840836255-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790512840836255-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45133
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790512840836255-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790512840836255-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:45:04.227530 21298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:45:04.228585 21298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:45:04.237373 21304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:45:04.237380 21306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:45:04.237380 21303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:45:04.237591 21298 server_base.cc:1034] running on GCE node
I20260927 12:45:04.238211 21298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:45:04.239267 21298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:45:04.240428 21298 hybrid_clock.cc:648] HybridClock initialized: now 1790513104240404 us; error 34 us; skew 500 ppm
Sep 27 12:45:04 dist-test-slave-bnc1 krb5kdc[21285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513104, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:45:04.243234 21298 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 12:45:04.244130 21298 webserver.cc:460] Webserver started at http://127.0.0.1:34581/ using document root <none> and password file <none>
I20260927 12:45:04.244697 21298 fs_manager.cc:359] Metadata directory not provided
I20260927 12:45:04.244757 21298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:45:04.244967 21298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:45:04.247049 21298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790512840836255-392-0/minicluster-data/master-0/data/instance:
uuid: "59c61c6fceba421c8c371320bc0da872"
format_stamp: "Formatted at 2026-09-27 12:45:04 on dist-test-slave-bnc1"
server_key: "bf9f07f1f2cfa47bac050170b2f986aa"
server_key_iv: "dd3a65927a5e5c46a95a5936409876d6"
server_key_version: "encryptionkey@0"
I20260927 12:45:04.247598 21298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790512840836255-392-0/minicluster-data/master-0/wal/instance:
uuid: "59c61c6fceba421c8c371320bc0da872"
format_stamp: "Formatted at 2026-09-27 12:45:04 on dist-test-slave-bnc1"
server_key: "bf9f07f1f2cfa47bac050170b2f986aa"
server_key_iv: "dd3a65927a5e5c46a95a5936409876d6"
server_key_version: "encryptionkey@0"
I20260927 12:45:04.251281 21298 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260927 12:45:04.253610 21313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:45:04.254930 21298 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 12:45:04.255108 21298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790512840836255-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790512840836255-392-0/minicluster-data/master-0/wal
uuid: "59c61c6fceba421c8c371320bc0da872"
format_stamp: "Formatted at 2026-09-27 12:45:04 on dist-test-slave-bnc1"
server_key: "bf9f07f1f2cfa47bac050170b2f986aa"
server_key_iv: "dd3a65927a5e5c46a95a5936409876d6"
server_key_version: "encryptionkey@0"
I20260927 12:45:04.255208 21298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790512840836255-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790512840836255-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790512840836255-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:45:04.269876 21298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:45:04.272687 21298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:45:04.272900 21298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:45:04.280154 21298 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45133
I20260927 12:45:04.280174 21365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45133 every 8 connection(s)
I20260927 12:45:04.281113 21298 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:45:04.283787 21366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:45:04.288751 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 21298
I20260927 12:45:04.288900 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790512840836255-392-0/minicluster-data/master-0/wal/instance
I20260927 12:45:04.289193 392 external_mini_cluster.cc:1428] Setting key 95b52ddbd8e58e51862f2b5a98d3ac80
I20260927 12:45:04.289208 21366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59c61c6fceba421c8c371320bc0da872: Bootstrap starting.
I20260927 12:45:04.291704 21366 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 59c61c6fceba421c8c371320bc0da872: Neither blocks nor log segments found. Creating new log.
I20260927 12:45:04.292431 21366 log.cc:824] T 00000000000000000000000000000000 P 59c61c6fceba421c8c371320bc0da872: Log is configured to *not* fsync() on all Append() calls
I20260927 12:45:04.294255 21366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59c61c6fceba421c8c371320bc0da872: No bootstrap required, opened a new log
Sep 27 12:45:04 dist-test-slave-bnc1 krb5kdc[21285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513104, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 12:45:04.297886 21366 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59c61c6fceba421c8c371320bc0da872 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59c61c6fceba421c8c371320bc0da872" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45133 } }
I20260927 12:45:04.298133 21366 raft_consensus.cc:374] T 00000000000000000000000000000000 P 59c61c6fceba421c8c371320bc0da872 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:45:04.298210 21366 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59c61c6fceba421c8c371320bc0da872 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59c61c6fceba421c8c371320bc0da872, State: Initialized, Role: FOLLOWER
I20260927 12:45:04.298739 21366 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59c61c6fceba421c8c371320bc0da872 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59c61c6fceba421c8c371320bc0da872" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45133 } }
I20260927 12:45:04.298894 21366 raft_consensus.cc:388] T 00000000000000000000000000000000 P 59c61c6fceba421c8c371320bc0da872 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:45:04.299034 21366 raft_consensus.cc:482] T 00000000000000000000000000000000 P 59c61c6fceba421c8c371320bc0da872 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:45:04.299156 21366 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59c61c6fceba421c8c371320bc0da872 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:45:04.299978 21366 raft_consensus.cc:504] T 00000000000000000000000000000000 P 59c61c6fceba421c8c371320bc0da872 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59c61c6fceba421c8c371320bc0da872" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45133 } }
I20260927 12:45:04.300316 21366 leader_election.cc:302] T 00000000000000000000000000000000 P 59c61c6fceba421c8c371320bc0da872 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 59c61c6fceba421c8c371320bc0da872; no voters:
I20260927 12:45:04.300670 21366 leader_election.cc:288] T 00000000000000000000000000000000 P 59c61c6fceba421c8c371320bc0da872 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:45:04.300745 21371 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 59c61c6fceba421c8c371320bc0da872 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:45:04.300944 21371 raft_consensus.cc:686] T 00000000000000000000000000000000 P 59c61c6fceba421c8c371320bc0da872 [term 1 LEADER]: Becoming Leader. State: Replica: 59c61c6fceba421c8c371320bc0da872, State: Running, Role: LEADER
I20260927 12:45:04.301287 21371 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59c61c6fceba421c8c371320bc0da872 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59c61c6fceba421c8c371320bc0da872" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45133 } }
I20260927 12:45:04.302013 21366 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59c61c6fceba421c8c371320bc0da872 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:45:04.302961 21373 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59c61c6fceba421c8c371320bc0da872 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "59c61c6fceba421c8c371320bc0da872" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59c61c6fceba421c8c371320bc0da872" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45133 } } }
I20260927 12:45:04.303361 21373 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59c61c6fceba421c8c371320bc0da872 [sys.catalog]: This master's current role is: LEADER
I20260927 12:45:04.303247 21372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59c61c6fceba421c8c371320bc0da872 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59c61c6fceba421c8c371320bc0da872. Latest consensus state: current_term: 1 leader_uuid: "59c61c6fceba421c8c371320bc0da872" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59c61c6fceba421c8c371320bc0da872" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45133 } } }
I20260927 12:45:04.303690 21372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59c61c6fceba421c8c371320bc0da872 [sys.catalog]: This master's current role is: LEADER
I20260927 12:45:04.304422 21369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:04.290599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.119:54053 (local address 127.0.0.1:45133)
0927 12:45:04.291018 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:04.291033 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:04.291075 (+ 42us) server_negotiation.cc:407] Connection header received
0927 12:45:04.291678 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:04.291693 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:04.292089 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:04.292418 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:04.293199 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:04.294213 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:04.295079 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:04.295533 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:04.295705 (+ 172us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:45:04.298381 (+ 2676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:45:04.298387 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:45:04.298402 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:45:04.298431 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:45:04.300591 (+ 2160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:45:04.301217 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:45:04.301223 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:45:04.301230 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:45:04.301366 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:45:04.301773 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:45:04.301777 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:45:04.301779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:45:04.302284 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:45:04.302499 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0927 12:45:04.303064 (+ 565us) server_negotiation.cc:299] Negotiation successful
0927 12:45:04.303346 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":129,"threads_started":1}
I20260927 12:45:04.304430 21381 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:45:04.307689 21381 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:45:04.308132 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 12:45:04.313650 21381 catalog_manager.cc:1347] Generated new cluster ID: ddaa32f79af04ca79280204451d3a8c8
I20260927 12:45:04.313778 21381 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:45:04.319661 21369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:04.310226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.119:45993 (local address 127.0.0.1:45133)
0927 12:45:04.310335 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:04.310340 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:04.310528 (+ 188us) server_negotiation.cc:407] Connection header received
0927 12:45:04.310631 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:04.310634 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:04.310685 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:04.310773 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:04.311556 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:04.312806 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:04.314019 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:04.314362 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:04.314471 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:45:04.316194 (+ 1723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:45:04.316197 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:45:04.316200 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:45:04.316226 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:45:04.317879 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:45:04.318410 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:45:04.318413 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:45:04.318414 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:45:04.318490 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:45:04.318828 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:45:04.318831 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:45:04.318833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:45:04.319015 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:45:04.319146 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 12:45:04.319416 (+ 270us) server_negotiation.cc:299] Negotiation successful
0927 12:45:04.319503 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 12:45:04.324530 21381 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:45:04.325719 21381 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:45:04.337006 21381 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 59c61c6fceba421c8c371320bc0da872: Generated new TSK 0
I20260927 12:45:04.337915 21381 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:45:04.348471 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 21298
W20260927 12:45:04.353829 21283 connection.cc:535] client connection to 127.0.0.1:45133 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45133): Connection reset by peer (error 104)
2026-09-27T12:45:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1422 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790512840836255-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:45:04 dist-test-slave-bnc1 krb5kdc[21397](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:45:04 dist-test-slave-bnc1 krb5kdc[21397](info): set up 2 sockets
Sep 27 12:45:04 dist-test-slave-bnc1 krb5kdc[21397](info): commencing operation
krb5kdc: starting...
W20260927 12:45:05.389876 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790512840836255-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790512840836255-392-0/krb5kdc/test-user.keytab.
Sep 27 12:45:05 dist-test-slave-bnc1 krb5kdc[21397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:45:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:45:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790512840836255-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790512840836255-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 12:45:05.513425 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790512840836255-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790512840836255-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790512840836255-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44607
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790512840836255-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44607
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:45:05.622807 21410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:45:05.623137 21410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:45:05.623230 21410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:45:05.627327 21410 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:45:05.627415 21410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:45:05.627454 21410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:45:05.627513 21410 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:45:05.627552 21410 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:45:05.632762 21410 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790512840836255-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790512840836255-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44607
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790512840836255-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790512840836255-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:45:05.633823 21410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:45:05.634851 21410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 12:45:05.643493 21410 server_base.cc:1034] running on GCE node
W20260927 12:45:05.643565 21415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:45:05.643644 21416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:45:05.643577 21418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:45:05.644308 21410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:45:05.645377 21410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:45:05.646588 21410 hybrid_clock.cc:648] HybridClock initialized: now 1790513105646559 us; error 40 us; skew 500 ppm
Sep 27 12:45:05 dist-test-slave-bnc1 krb5kdc[21397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513105, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:45:05.649333 21410 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 12:45:05.650205 21410 webserver.cc:460] Webserver started at http://127.0.0.1:46155/ using document root <none> and password file <none>
I20260927 12:45:05.650744 21410 fs_manager.cc:359] Metadata directory not provided
I20260927 12:45:05.650831 21410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:45:05.651054 21410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:45:05.652966 21410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790512840836255-392-0/minicluster-data/master-0/data/instance:
uuid: "d9db4ca8da974c108cd044f269015d5e"
format_stamp: "Formatted at 2026-09-27 12:45:05 on dist-test-slave-bnc1"
server_key: "6acdacf8b7e0bcbe6ced6a0474ef365f"
server_key_iv: "80616f68a5de6df1edad10a30e09399e"
server_key_version: "encryptionkey@0"
I20260927 12:45:05.653501 21410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790512840836255-392-0/minicluster-data/master-0/wal/instance:
uuid: "d9db4ca8da974c108cd044f269015d5e"
format_stamp: "Formatted at 2026-09-27 12:45:05 on dist-test-slave-bnc1"
server_key: "6acdacf8b7e0bcbe6ced6a0474ef365f"
server_key_iv: "80616f68a5de6df1edad10a30e09399e"
server_key_version: "encryptionkey@0"
I20260927 12:45:05.656827 21410 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 12:45:05.659149 21425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:45:05.660288 21410 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 12:45:05.660426 21410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790512840836255-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790512840836255-392-0/minicluster-data/master-0/wal
uuid: "d9db4ca8da974c108cd044f269015d5e"
format_stamp: "Formatted at 2026-09-27 12:45:05 on dist-test-slave-bnc1"
server_key: "6acdacf8b7e0bcbe6ced6a0474ef365f"
server_key_iv: "80616f68a5de6df1edad10a30e09399e"
server_key_version: "encryptionkey@0"
I20260927 12:45:05.660528 21410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790512840836255-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790512840836255-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790512840836255-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:45:05.685480 21410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:45:05.688300 21410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:45:05.688496 21410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:45:05.695235 21410 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44607
I20260927 12:45:05.695250 21477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44607 every 8 connection(s)
I20260927 12:45:05.696185 21410 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:45:05.698890 21478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:45:05.699517 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 21410
I20260927 12:45:05.699669 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790512840836255-392-0/minicluster-data/master-0/wal/instance
I20260927 12:45:05.699950 392 external_mini_cluster.cc:1428] Setting key 40e786d29dca969446c7402e5ec51c75
I20260927 12:45:05.705425 21478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9db4ca8da974c108cd044f269015d5e: Bootstrap starting.
I20260927 12:45:05.707937 21478 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d9db4ca8da974c108cd044f269015d5e: Neither blocks nor log segments found. Creating new log.
I20260927 12:45:05.708801 21478 log.cc:824] T 00000000000000000000000000000000 P d9db4ca8da974c108cd044f269015d5e: Log is configured to *not* fsync() on all Append() calls
I20260927 12:45:05.709944 21481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:05.701375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.119:58647 (local address 127.0.0.1:44607)
0927 12:45:05.701816 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:05.701835 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:05.701880 (+ 45us) server_negotiation.cc:407] Connection header received
0927 12:45:05.702692 (+ 812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:05.702714 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:05.703145 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:05.703502 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:05.703898 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:05.705329 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:05.706329 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:05.707014 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:05.707179 (+ 165us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:45:05.708006 (+ 827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:45:05.708013 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:45:05.708029 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:45:05.708522 (+ 493us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:45:05.708589 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:45:05.708619 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:45:05.708751 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 12:45:05.708943 (+ 192us) server_negotiation.cc:299] Negotiation successful
0927 12:45:05.709070 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":156,"threads_started":1}
I20260927 12:45:05.711014 21478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9db4ca8da974c108cd044f269015d5e: No bootstrap required, opened a new log
I20260927 12:45:05.713094 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 12:45:05.715236 21478 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9db4ca8da974c108cd044f269015d5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9db4ca8da974c108cd044f269015d5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44607 } }
I20260927 12:45:05.715521 21478 raft_consensus.cc:374] T 00000000000000000000000000000000 P d9db4ca8da974c108cd044f269015d5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:45:05.715615 21478 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9db4ca8da974c108cd044f269015d5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9db4ca8da974c108cd044f269015d5e, State: Initialized, Role: FOLLOWER
I20260927 12:45:05.716073 21478 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9db4ca8da974c108cd044f269015d5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9db4ca8da974c108cd044f269015d5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44607 } }
I20260927 12:45:05.716223 21478 raft_consensus.cc:388] T 00000000000000000000000000000000 P d9db4ca8da974c108cd044f269015d5e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:45:05.716308 21478 raft_consensus.cc:482] T 00000000000000000000000000000000 P d9db4ca8da974c108cd044f269015d5e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:45:05.716404 21478 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9db4ca8da974c108cd044f269015d5e [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:45:05.717331 21478 raft_consensus.cc:504] T 00000000000000000000000000000000 P d9db4ca8da974c108cd044f269015d5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9db4ca8da974c108cd044f269015d5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44607 } }
I20260927 12:45:05.717728 21478 leader_election.cc:302] T 00000000000000000000000000000000 P d9db4ca8da974c108cd044f269015d5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d9db4ca8da974c108cd044f269015d5e; no voters:
I20260927 12:45:05.718063 21478 leader_election.cc:288] T 00000000000000000000000000000000 P d9db4ca8da974c108cd044f269015d5e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:45:05.718178 21488 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d9db4ca8da974c108cd044f269015d5e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:45:05.718374 21488 raft_consensus.cc:686] T 00000000000000000000000000000000 P d9db4ca8da974c108cd044f269015d5e [term 1 LEADER]: Becoming Leader. State: Replica: d9db4ca8da974c108cd044f269015d5e, State: Running, Role: LEADER
I20260927 12:45:05.718668 21488 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9db4ca8da974c108cd044f269015d5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9db4ca8da974c108cd044f269015d5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44607 } }
I20260927 12:45:05.719343 21478 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9db4ca8da974c108cd044f269015d5e [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:45:05.719527 21481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:05.715203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.119:55573 (local address 127.0.0.1:44607)
0927 12:45:05.715323 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:05.715329 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:05.715495 (+ 166us) server_negotiation.cc:407] Connection header received
0927 12:45:05.715637 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:05.715642 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:05.715709 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:05.715788 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:05.716066 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:05.716757 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:05.717617 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:05.717922 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:05.718071 (+ 149us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:45:05.719059 (+ 988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:45:05.719063 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:45:05.719066 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:45:05.719077 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:45:05.719110 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:45:05.719121 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:45:05.719216 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 12:45:05.719334 (+ 118us) server_negotiation.cc:299] Negotiation successful
0927 12:45:05.719367 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 12:45:05.719802 21488 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9db4ca8da974c108cd044f269015d5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9db4ca8da974c108cd044f269015d5e. Latest consensus state: current_term: 1 leader_uuid: "d9db4ca8da974c108cd044f269015d5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9db4ca8da974c108cd044f269015d5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44607 } } }
I20260927 12:45:05.719911 21488 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9db4ca8da974c108cd044f269015d5e [sys.catalog]: This master's current role is: LEADER
I20260927 12:45:05.720206 21490 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9db4ca8da974c108cd044f269015d5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d9db4ca8da974c108cd044f269015d5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9db4ca8da974c108cd044f269015d5e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44607 } } }
I20260927 12:45:05.720293 21490 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9db4ca8da974c108cd044f269015d5e [sys.catalog]: This master's current role is: LEADER
I20260927 12:45:05.720593 21492 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:45:05.723762 21492 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:45:05.729666 21492 catalog_manager.cc:1347] Generated new cluster ID: aa0545d81114475aa13e43776d51f2b9
I20260927 12:45:05.729750 21492 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:45:05.742412 21492 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:45:05.743690 21492 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:45:05.748378 21492 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d9db4ca8da974c108cd044f269015d5e: Generated new TSK 0
I20260927 12:45:05.749172 21492 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:45:05.756925 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 21410
W20260927 12:45:05.762312 21395 connection.cc:535] client connection to 127.0.0.1:44607 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44607): Connection reset by peer (error 104)
2026-09-27T12:45:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1408 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790512840836255-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:45:05 dist-test-slave-bnc1 krb5kdc[21509](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:45:05 dist-test-slave-bnc1 krb5kdc[21509](info): set up 2 sockets
Sep 27 12:45:05 dist-test-slave-bnc1 krb5kdc[21509](info): commencing operation
krb5kdc: starting...
W20260927 12:45:06.801798 392 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790512840836255-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790512840836255-392-0/krb5kdc/test-user.keytab.
Sep 27 12:45:06 dist-test-slave-bnc1 krb5kdc[21509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:45:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:45:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790512840836255-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790512840836255-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 12:45:06.920766 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790512840836255-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790512840836255-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790512840836255-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34621
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790512840836255-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34621
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:45:07.028990 21522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:45:07.029321 21522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:45:07.029367 21522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:45:07.033308 21522 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:45:07.033368 21522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:45:07.033388 21522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:45:07.033404 21522 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:45:07.033419 21522 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:45:07.038671 21522 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790512840836255-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790512840836255-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34621
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790512840836255-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21522
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790512840836255-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:45:07.039719 21522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:45:07.040732 21522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:45:07.048951 21528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:45:07.049002 21530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:45:07.049011 21527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:45:07.049096 21522 server_base.cc:1034] running on GCE node
I20260927 12:45:07.049872 21522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:45:07.050973 21522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:45:07.052139 21522 hybrid_clock.cc:648] HybridClock initialized: now 1790513107052104 us; error 42 us; skew 500 ppm
Sep 27 12:45:07 dist-test-slave-bnc1 krb5kdc[21509](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513107, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:45:07.054809 21522 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 12:45:07.055702 21522 webserver.cc:460] Webserver started at http://127.0.0.1:35623/ using document root <none> and password file <none>
I20260927 12:45:07.056250 21522 fs_manager.cc:359] Metadata directory not provided
I20260927 12:45:07.056341 21522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:45:07.056564 21522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:45:07.058643 21522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790512840836255-392-0/minicluster-data/master-0/data/instance:
uuid: "2aab73940b8b437ca02d40da2442ee61"
format_stamp: "Formatted at 2026-09-27 12:45:07 on dist-test-slave-bnc1"
server_key: "c1766ea477d91b237f9d6d419ff2ecb0"
server_key_iv: "e7b116125edf097d37629e11f6b77d29"
server_key_version: "encryptionkey@0"
I20260927 12:45:07.059207 21522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790512840836255-392-0/minicluster-data/master-0/wal/instance:
uuid: "2aab73940b8b437ca02d40da2442ee61"
format_stamp: "Formatted at 2026-09-27 12:45:07 on dist-test-slave-bnc1"
server_key: "c1766ea477d91b237f9d6d419ff2ecb0"
server_key_iv: "e7b116125edf097d37629e11f6b77d29"
server_key_version: "encryptionkey@0"
I20260927 12:45:07.062749 21522 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 12:45:07.065028 21537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:45:07.066252 21522 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 12:45:07.066388 21522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790512840836255-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790512840836255-392-0/minicluster-data/master-0/wal
uuid: "2aab73940b8b437ca02d40da2442ee61"
format_stamp: "Formatted at 2026-09-27 12:45:07 on dist-test-slave-bnc1"
server_key: "c1766ea477d91b237f9d6d419ff2ecb0"
server_key_iv: "e7b116125edf097d37629e11f6b77d29"
server_key_version: "encryptionkey@0"
I20260927 12:45:07.066494 21522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790512840836255-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790512840836255-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790512840836255-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:45:07.070953 21522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:45:07.073791 21522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:45:07.073975 21522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:45:07.080916 21522 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34621
I20260927 12:45:07.080936 21589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34621 every 8 connection(s)
I20260927 12:45:07.081909 21522 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:45:07.084589 21590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:45:07.086418 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 21522
I20260927 12:45:07.086546 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790512840836255-392-0/minicluster-data/master-0/wal/instance
I20260927 12:45:07.086846 392 external_mini_cluster.cc:1428] Setting key eb5c448e5df3310955b7476bb5d8c69a
I20260927 12:45:07.091096 21590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2aab73940b8b437ca02d40da2442ee61: Bootstrap starting.
I20260927 12:45:07.092038 21593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:07.088247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.119:40401 (local address 127.0.0.1:34621)
0927 12:45:07.088688 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:07.088702 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:07.088737 (+ 35us) server_negotiation.cc:407] Connection header received
0927 12:45:07.089357 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:07.089374 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:07.089725 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:07.090065 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:07.090854 (+ 789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:45:07.090858 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:45:07.090870 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:45:07.090880 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:45:07.090932 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:45:07.090950 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:45:07.091099 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 12:45:07.091171 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 12:45:07.091336 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":145,"threads_started":1}
I20260927 12:45:07.093340 21590 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2aab73940b8b437ca02d40da2442ee61: Neither blocks nor log segments found. Creating new log.
I20260927 12:45:07.094220 21590 log.cc:824] T 00000000000000000000000000000000 P 2aab73940b8b437ca02d40da2442ee61: Log is configured to *not* fsync() on all Append() calls
I20260927 12:45:07.094832 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 12:45:07.096460 21590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2aab73940b8b437ca02d40da2442ee61: No bootstrap required, opened a new log
I20260927 12:45:07.098591 21593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:07.097011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.119:47169 (local address 127.0.0.1:34621)
0927 12:45:07.097164 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:07.097169 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:07.097292 (+ 123us) server_negotiation.cc:407] Connection header received
0927 12:45:07.097423 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:07.097425 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:07.097451 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:07.097509 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:07.098201 (+ 692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:45:07.098204 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:45:07.098206 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:45:07.098216 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:45:07.098238 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:45:07.098247 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:45:07.098315 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 12:45:07.098417 (+ 102us) server_negotiation.cc:299] Negotiation successful
0927 12:45:07.098448 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 12:45:07.100907 21590 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2aab73940b8b437ca02d40da2442ee61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aab73940b8b437ca02d40da2442ee61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34621 } }
I20260927 12:45:07.101150 21590 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2aab73940b8b437ca02d40da2442ee61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:45:07.101207 21590 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2aab73940b8b437ca02d40da2442ee61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aab73940b8b437ca02d40da2442ee61, State: Initialized, Role: FOLLOWER
I20260927 12:45:07.101636 21590 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2aab73940b8b437ca02d40da2442ee61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aab73940b8b437ca02d40da2442ee61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34621 } }
I20260927 12:45:07.101788 21590 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2aab73940b8b437ca02d40da2442ee61 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:45:07.101858 21590 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2aab73940b8b437ca02d40da2442ee61 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:45:07.101970 21590 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2aab73940b8b437ca02d40da2442ee61 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:45:07.102957 21590 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2aab73940b8b437ca02d40da2442ee61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aab73940b8b437ca02d40da2442ee61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34621 } }
I20260927 12:45:07.103291 21590 leader_election.cc:302] T 00000000000000000000000000000000 P 2aab73940b8b437ca02d40da2442ee61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2aab73940b8b437ca02d40da2442ee61; no voters:
I20260927 12:45:07.103593 21590 leader_election.cc:288] T 00000000000000000000000000000000 P 2aab73940b8b437ca02d40da2442ee61 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:45:07.103791 21600 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2aab73940b8b437ca02d40da2442ee61 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:45:07.104079 21600 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2aab73940b8b437ca02d40da2442ee61 [term 1 LEADER]: Becoming Leader. State: Replica: 2aab73940b8b437ca02d40da2442ee61, State: Running, Role: LEADER
I20260927 12:45:07.104432 21600 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2aab73940b8b437ca02d40da2442ee61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aab73940b8b437ca02d40da2442ee61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34621 } }
I20260927 12:45:07.104794 21590 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2aab73940b8b437ca02d40da2442ee61 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:45:07.105988 21601 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2aab73940b8b437ca02d40da2442ee61 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2aab73940b8b437ca02d40da2442ee61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aab73940b8b437ca02d40da2442ee61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34621 } } }
I20260927 12:45:07.106142 21601 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2aab73940b8b437ca02d40da2442ee61 [sys.catalog]: This master's current role is: LEADER
I20260927 12:45:07.106506 21602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2aab73940b8b437ca02d40da2442ee61 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2aab73940b8b437ca02d40da2442ee61. Latest consensus state: current_term: 1 leader_uuid: "2aab73940b8b437ca02d40da2442ee61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aab73940b8b437ca02d40da2442ee61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34621 } } }
I20260927 12:45:07.106583 21602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2aab73940b8b437ca02d40da2442ee61 [sys.catalog]: This master's current role is: LEADER
I20260927 12:45:07.106863 21609 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:45:07.109797 21609 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:45:07.115177 21609 catalog_manager.cc:1347] Generated new cluster ID: 0fefedbcc3b14b4c8e56cc467295e485
I20260927 12:45:07.115257 21609 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:45:07.138306 21609 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:45:07.138670 21609 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:45:07.144196 21609 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2aab73940b8b437ca02d40da2442ee61: Generated new TSK 0
I20260927 12:45:07.145021 21609 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:45:07.158528 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 21522
2026-09-27T12:45:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1401 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790512840836255-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:45:07 dist-test-slave-bnc1 krb5kdc[21621](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:45:07 dist-test-slave-bnc1 krb5kdc[21621](info): set up 2 sockets
Sep 27 12:45:07 dist-test-slave-bnc1 krb5kdc[21621](info): commencing operation
krb5kdc: starting...
W20260927 12:45:08.189764 392 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790512840836255-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790512840836255-392-0/krb5kdc/test-user.keytab.
Sep 27 12:45:08 dist-test-slave-bnc1 krb5kdc[21621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513108, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:45:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:45:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790512840836255-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790512840836255-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 12:45:08.313164 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790512840836255-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790512840836255-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790512840836255-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39871
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790512840836255-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39871
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:45:08.423344 21634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:45:08.423635 21634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:45:08.423677 21634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:45:08.427626 21634 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:45:08.427693 21634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:45:08.427712 21634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:45:08.427728 21634 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:45:08.427743 21634 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:45:08.433699 21634 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790512840836255-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790512840836255-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39871
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790512840836255-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790512840836255-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:45:08.434857 21634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:45:08.435860 21634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:45:08.444777 21640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:45:08.444768 21639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:45:08.444988 21634 server_base.cc:1034] running on GCE node
W20260927 12:45:08.444768 21642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:45:08.445736 21634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:45:08.446867 21634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:45:08.448022 21634 hybrid_clock.cc:648] HybridClock initialized: now 1790513108447986 us; error 53 us; skew 500 ppm
Sep 27 12:45:08 dist-test-slave-bnc1 krb5kdc[21621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513108, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:45:08.455744 21634 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 12:45:08.456737 21634 webserver.cc:460] Webserver started at http://127.0.0.1:46701/ using document root <none> and password file <none>
I20260927 12:45:08.457304 21634 fs_manager.cc:359] Metadata directory not provided
I20260927 12:45:08.457366 21634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:45:08.457674 21634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:45:08.459734 21634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790512840836255-392-0/minicluster-data/master-0/data/instance:
uuid: "7b27b34fdd3442e8998015f445934aaa"
format_stamp: "Formatted at 2026-09-27 12:45:08 on dist-test-slave-bnc1"
server_key: "6e72584efc2cd7a73b6d19fdaeca665a"
server_key_iv: "36dbe29a57afdb73d7bec7c602345b46"
server_key_version: "encryptionkey@0"
I20260927 12:45:08.460361 21634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790512840836255-392-0/minicluster-data/master-0/wal/instance:
uuid: "7b27b34fdd3442e8998015f445934aaa"
format_stamp: "Formatted at 2026-09-27 12:45:08 on dist-test-slave-bnc1"
server_key: "6e72584efc2cd7a73b6d19fdaeca665a"
server_key_iv: "36dbe29a57afdb73d7bec7c602345b46"
server_key_version: "encryptionkey@0"
I20260927 12:45:08.464054 21634 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 12:45:08.466507 21649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:45:08.467716 21634 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 12:45:08.467869 21634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790512840836255-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790512840836255-392-0/minicluster-data/master-0/wal
uuid: "7b27b34fdd3442e8998015f445934aaa"
format_stamp: "Formatted at 2026-09-27 12:45:08 on dist-test-slave-bnc1"
server_key: "6e72584efc2cd7a73b6d19fdaeca665a"
server_key_iv: "36dbe29a57afdb73d7bec7c602345b46"
server_key_version: "encryptionkey@0"
I20260927 12:45:08.467990 21634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790512840836255-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790512840836255-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790512840836255-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:45:08.494928 21634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:45:08.498330 21634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:45:08.498548 21634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:45:08.505844 21634 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39871
I20260927 12:45:08.505865 21701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39871 every 8 connection(s)
I20260927 12:45:08.506848 21634 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:45:08.509677 21702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:45:08.509869 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 21634
I20260927 12:45:08.510017 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790512840836255-392-0/minicluster-data/master-0/wal/instance
I20260927 12:45:08.510320 392 external_mini_cluster.cc:1428] Setting key 44587264d606fd8d114733d784e04c70
I20260927 12:45:08.516764 21702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b27b34fdd3442e8998015f445934aaa: Bootstrap starting.
I20260927 12:45:08.519200 21702 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b27b34fdd3442e8998015f445934aaa: Neither blocks nor log segments found. Creating new log.
Sep 27 12:45:08 dist-test-slave-bnc1 krb5kdc[21621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513108, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 12:45:08.520071 21702 log.cc:824] T 00000000000000000000000000000000 P 7b27b34fdd3442e8998015f445934aaa: Log is configured to *not* fsync() on all Append() calls
I20260927 12:45:08.522004 21702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b27b34fdd3442e8998015f445934aaa: No bootstrap required, opened a new log
I20260927 12:45:08.524720 21705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:08.511739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.119:47683 (local address 127.0.0.1:39871)
0927 12:45:08.512147 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:08.512166 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:08.512208 (+ 42us) server_negotiation.cc:407] Connection header received
0927 12:45:08.513020 (+ 812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:08.513044 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:08.513504 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:08.513896 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:08.514557 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:08.516060 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:08.516905 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:08.517389 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:08.517602 (+ 213us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:45:08.520151 (+ 2549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:45:08.520156 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:45:08.520169 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:45:08.520219 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:45:08.521994 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:45:08.522513 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:45:08.522518 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:45:08.522522 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:45:08.522633 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:45:08.522962 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:45:08.522965 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:45:08.522967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:45:08.523335 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:45:08.523546 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0927 12:45:08.523818 (+ 272us) server_negotiation.cc:299] Negotiation successful
0927 12:45:08.524022 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":117,"threads_started":1}
I20260927 12:45:08.525780 21702 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b27b34fdd3442e8998015f445934aaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b27b34fdd3442e8998015f445934aaa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39871 } }
I20260927 12:45:08.526048 21702 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b27b34fdd3442e8998015f445934aaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:45:08.526116 21702 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b27b34fdd3442e8998015f445934aaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b27b34fdd3442e8998015f445934aaa, State: Initialized, Role: FOLLOWER
I20260927 12:45:08.526559 21702 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b27b34fdd3442e8998015f445934aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b27b34fdd3442e8998015f445934aaa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39871 } }
I20260927 12:45:08.526732 21702 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b27b34fdd3442e8998015f445934aaa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:45:08.526803 21702 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b27b34fdd3442e8998015f445934aaa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:45:08.526901 21702 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b27b34fdd3442e8998015f445934aaa [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:45:08.527777 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 12:45:08.528036 21702 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b27b34fdd3442e8998015f445934aaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b27b34fdd3442e8998015f445934aaa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39871 } }
I20260927 12:45:08.528365 21702 leader_election.cc:302] T 00000000000000000000000000000000 P 7b27b34fdd3442e8998015f445934aaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b27b34fdd3442e8998015f445934aaa; no voters:
I20260927 12:45:08.528682 21702 leader_election.cc:288] T 00000000000000000000000000000000 P 7b27b34fdd3442e8998015f445934aaa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:45:08.528748 21707 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b27b34fdd3442e8998015f445934aaa [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:45:08.528940 21707 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b27b34fdd3442e8998015f445934aaa [term 1 LEADER]: Becoming Leader. State: Replica: 7b27b34fdd3442e8998015f445934aaa, State: Running, Role: LEADER
I20260927 12:45:08.529238 21707 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b27b34fdd3442e8998015f445934aaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b27b34fdd3442e8998015f445934aaa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39871 } }
I20260927 12:45:08.529942 21702 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b27b34fdd3442e8998015f445934aaa [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:45:08.530371 21711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b27b34fdd3442e8998015f445934aaa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b27b34fdd3442e8998015f445934aaa. Latest consensus state: current_term: 1 leader_uuid: "7b27b34fdd3442e8998015f445934aaa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b27b34fdd3442e8998015f445934aaa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39871 } } }
I20260927 12:45:08.530395 21710 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b27b34fdd3442e8998015f445934aaa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b27b34fdd3442e8998015f445934aaa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b27b34fdd3442e8998015f445934aaa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39871 } } }
I20260927 12:45:08.530543 21711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b27b34fdd3442e8998015f445934aaa [sys.catalog]: This master's current role is: LEADER
I20260927 12:45:08.530546 21710 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b27b34fdd3442e8998015f445934aaa [sys.catalog]: This master's current role is: LEADER
I20260927 12:45:08.530990 21716 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:45:08.534399 21716 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:45:08.539016 21705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:08.530105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.119:53977 (local address 127.0.0.1:39871)
0927 12:45:08.530195 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:08.530199 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:08.530414 (+ 215us) server_negotiation.cc:407] Connection header received
0927 12:45:08.530624 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:08.530627 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:08.530674 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:08.530751 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:08.531674 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:08.532391 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:08.533550 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:08.533832 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:08.533951 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:45:08.535692 (+ 1741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:45:08.535696 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:45:08.535698 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:45:08.535723 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:45:08.537189 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:45:08.537738 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:45:08.537741 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:45:08.537743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:45:08.537826 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:45:08.538153 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:45:08.538157 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:45:08.538158 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:45:08.538340 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:45:08.538488 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 12:45:08.538759 (+ 271us) server_negotiation.cc:299] Negotiation successful
0927 12:45:08.538851 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 12:45:08.542821 21716 catalog_manager.cc:1347] Generated new cluster ID: a17095372c144e299cd821b7fdcf921f
I20260927 12:45:08.542888 21716 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:45:08.559324 21716 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:45:08.560302 21716 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:45:08.573244 21716 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b27b34fdd3442e8998015f445934aaa: Generated new TSK 0
I20260927 12:45:08.574002 21716 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:45:08.585873 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 21634
W20260927 12:45:08.591296 21619 connection.cc:535] client connection to 127.0.0.1:39871 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39871): Connection reset by peer (error 104)
2026-09-27T12:45:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1426 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790512840836255-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:45:08 dist-test-slave-bnc1 krb5kdc[21733](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:45:08 dist-test-slave-bnc1 krb5kdc[21733](info): set up 2 sockets
Sep 27 12:45:08 dist-test-slave-bnc1 krb5kdc[21733](info): commencing operation
krb5kdc: starting...
W20260927 12:45:09.653820 392 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790512840836255-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790512840836255-392-0/krb5kdc/test-user.keytab.
Sep 27 12:45:09 dist-test-slave-bnc1 krb5kdc[21733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:45:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:45:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790512840836255-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790512840836255-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 12:45:09.772316 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790512840836255-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790512840836255-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790512840836255-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45031
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790512840836255-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45031
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:45:09.883148 21746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:45:09.883471 21746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:45:09.883553 21746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:45:09.887669 21746 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:45:09.887756 21746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:45:09.887794 21746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:45:09.887852 21746 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:45:09.887890 21746 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:45:09.893165 21746 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790512840836255-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790512840836255-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45031
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790512840836255-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21746
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790512840836255-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:45:09.894294 21746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:45:09.895399 21746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:45:09.904119 21754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:45:09.904093 21751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:45:09.904209 21746 server_base.cc:1034] running on GCE node
W20260927 12:45:09.904093 21752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:45:09.905015 21746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:45:09.906344 21746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:45:09.907528 21746 hybrid_clock.cc:648] HybridClock initialized: now 1790513109907488 us; error 54 us; skew 500 ppm
Sep 27 12:45:09 dist-test-slave-bnc1 krb5kdc[21733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513109, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:45:09.910333 21746 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 12:45:09.911284 21746 webserver.cc:460] Webserver started at http://127.0.0.1:40049/ using document root <none> and password file <none>
I20260927 12:45:09.911907 21746 fs_manager.cc:359] Metadata directory not provided
I20260927 12:45:09.912007 21746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:45:09.912246 21746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:45:09.914425 21746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790512840836255-392-0/minicluster-data/master-0/data/instance:
uuid: "13dec0db3dba4c9ca280327b398eb000"
format_stamp: "Formatted at 2026-09-27 12:45:09 on dist-test-slave-bnc1"
server_key: "ee1e397055d3952809667857ddc6fbd5"
server_key_iv: "c1ee92913044e784e2ba984ba732a3a7"
server_key_version: "encryptionkey@0"
I20260927 12:45:09.915001 21746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790512840836255-392-0/minicluster-data/master-0/wal/instance:
uuid: "13dec0db3dba4c9ca280327b398eb000"
format_stamp: "Formatted at 2026-09-27 12:45:09 on dist-test-slave-bnc1"
server_key: "ee1e397055d3952809667857ddc6fbd5"
server_key_iv: "c1ee92913044e784e2ba984ba732a3a7"
server_key_version: "encryptionkey@0"
I20260927 12:45:09.918815 21746 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 12:45:09.921392 21761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:45:09.922708 21746 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 12:45:09.922855 21746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790512840836255-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790512840836255-392-0/minicluster-data/master-0/wal
uuid: "13dec0db3dba4c9ca280327b398eb000"
format_stamp: "Formatted at 2026-09-27 12:45:09 on dist-test-slave-bnc1"
server_key: "ee1e397055d3952809667857ddc6fbd5"
server_key_iv: "c1ee92913044e784e2ba984ba732a3a7"
server_key_version: "encryptionkey@0"
I20260927 12:45:09.922974 21746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790512840836255-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790512840836255-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790512840836255-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:45:09.939456 21746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:45:09.942298 21746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:45:09.942502 21746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:45:09.949573 21746 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45031
I20260927 12:45:09.949628 21813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45031 every 8 connection(s)
I20260927 12:45:09.950582 21746 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:45:09.953958 21814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:45:09.958225 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 21746
I20260927 12:45:09.958359 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790512840836255-392-0/minicluster-data/master-0/wal/instance
I20260927 12:45:09.958636 392 external_mini_cluster.cc:1428] Setting key c434135a7ff9bf02234c527df7ecd1ff
I20260927 12:45:09.961966 21814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13dec0db3dba4c9ca280327b398eb000: Bootstrap starting.
I20260927 12:45:09.964435 21814 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 13dec0db3dba4c9ca280327b398eb000: Neither blocks nor log segments found. Creating new log.
I20260927 12:45:09.965291 21814 log.cc:824] T 00000000000000000000000000000000 P 13dec0db3dba4c9ca280327b398eb000: Log is configured to *not* fsync() on all Append() calls
I20260927 12:45:09.967775 21814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13dec0db3dba4c9ca280327b398eb000: No bootstrap required, opened a new log
I20260927 12:45:09.968397 21817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:09.960624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.119:57473 (local address 127.0.0.1:45031)
0927 12:45:09.961137 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:09.961153 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:09.961189 (+ 36us) server_negotiation.cc:407] Connection header received
0927 12:45:09.961889 (+ 700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:09.961907 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:09.962306 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:09.962595 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:09.962911 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:09.963871 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:09.965069 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:09.965641 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:09.965818 (+ 177us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:45:09.966663 (+ 845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:45:09.966667 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:45:09.966680 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:45:09.967053 (+ 373us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:45:09.967113 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:45:09.967136 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:45:09.967238 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 12:45:09.967437 (+ 199us) server_negotiation.cc:299] Negotiation successful
0927 12:45:09.967619 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":134,"threads_started":1}
I20260927 12:45:09.971443 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 12:45:09.971926 21814 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13dec0db3dba4c9ca280327b398eb000 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13dec0db3dba4c9ca280327b398eb000" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45031 } }
I20260927 12:45:09.972206 21814 raft_consensus.cc:374] T 00000000000000000000000000000000 P 13dec0db3dba4c9ca280327b398eb000 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:45:09.972275 21814 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13dec0db3dba4c9ca280327b398eb000 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13dec0db3dba4c9ca280327b398eb000, State: Initialized, Role: FOLLOWER
I20260927 12:45:09.972748 21814 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13dec0db3dba4c9ca280327b398eb000 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13dec0db3dba4c9ca280327b398eb000" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45031 } }
I20260927 12:45:09.972931 21814 raft_consensus.cc:388] T 00000000000000000000000000000000 P 13dec0db3dba4c9ca280327b398eb000 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:45:09.973002 21814 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13dec0db3dba4c9ca280327b398eb000 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:45:09.973099 21814 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13dec0db3dba4c9ca280327b398eb000 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:45:09.974313 21814 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13dec0db3dba4c9ca280327b398eb000 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13dec0db3dba4c9ca280327b398eb000" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45031 } }
I20260927 12:45:09.974668 21814 leader_election.cc:302] T 00000000000000000000000000000000 P 13dec0db3dba4c9ca280327b398eb000 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 13dec0db3dba4c9ca280327b398eb000; no voters:
I20260927 12:45:09.974984 21814 leader_election.cc:288] T 00000000000000000000000000000000 P 13dec0db3dba4c9ca280327b398eb000 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:45:09.975075 21824 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13dec0db3dba4c9ca280327b398eb000 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:45:09.975343 21824 raft_consensus.cc:686] T 00000000000000000000000000000000 P 13dec0db3dba4c9ca280327b398eb000 [term 1 LEADER]: Becoming Leader. State: Replica: 13dec0db3dba4c9ca280327b398eb000, State: Running, Role: LEADER
I20260927 12:45:09.975711 21824 consensus_queue.cc:237] T 00000000000000000000000000000000 P 13dec0db3dba4c9ca280327b398eb000 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13dec0db3dba4c9ca280327b398eb000" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45031 } }
I20260927 12:45:09.976137 21814 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13dec0db3dba4c9ca280327b398eb000 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:45:09.977268 21825 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13dec0db3dba4c9ca280327b398eb000 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "13dec0db3dba4c9ca280327b398eb000" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13dec0db3dba4c9ca280327b398eb000" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45031 } } }
I20260927 12:45:09.977407 21825 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13dec0db3dba4c9ca280327b398eb000 [sys.catalog]: This master's current role is: LEADER
I20260927 12:45:09.977692 21825 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13dec0db3dba4c9ca280327b398eb000 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13dec0db3dba4c9ca280327b398eb000. Latest consensus state: current_term: 1 leader_uuid: "13dec0db3dba4c9ca280327b398eb000" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13dec0db3dba4c9ca280327b398eb000" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45031 } } }
I20260927 12:45:09.977833 21825 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13dec0db3dba4c9ca280327b398eb000 [sys.catalog]: This master's current role is: LEADER
I20260927 12:45:09.978505 21817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:09.973658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.119:59367 (local address 127.0.0.1:45031)
0927 12:45:09.973790 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:09.973794 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:09.973993 (+ 199us) server_negotiation.cc:407] Connection header received
0927 12:45:09.974112 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:09.974114 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:09.974160 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:09.974221 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:09.974514 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:09.975192 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:09.976285 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:09.976606 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:09.976697 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:45:09.977946 (+ 1249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:45:09.977950 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:45:09.977953 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:45:09.977967 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:45:09.977995 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:45:09.978008 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:45:09.978079 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 12:45:09.978339 (+ 260us) server_negotiation.cc:299] Negotiation successful
0927 12:45:09.978373 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 12:45:09.978845 21833 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:45:09.981491 21833 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:45:09.986958 21833 catalog_manager.cc:1347] Generated new cluster ID: 59386f63458342a796511c08ea0f9873
I20260927 12:45:09.987035 21833 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:45:10.013146 21833 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:45:10.014312 21833 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:45:10.019019 21833 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 13dec0db3dba4c9ca280327b398eb000: Generated new TSK 0
I20260927 12:45:10.019737 21833 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:45:10.027338 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 21746
W20260927 12:45:10.032644 21731 connection.cc:535] client connection to 127.0.0.1:45031 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45031): Connection reset by peer (error 104)
2026-09-27T12:45:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1441 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790512840836255-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:45:10 dist-test-slave-bnc1 krb5kdc[21845](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:45:10 dist-test-slave-bnc1 krb5kdc[21845](info): set up 2 sockets
Sep 27 12:45:10 dist-test-slave-bnc1 krb5kdc[21845](info): commencing operation
krb5kdc: starting...
W20260927 12:45:11.049721 392 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790512840836255-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790512840836255-392-0/krb5kdc/test-user.keytab.
Sep 27 12:45:11 dist-test-slave-bnc1 krb5kdc[21845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:45:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:45:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790512840836255-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790512840836255-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 12:45:11.170394 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790512840836255-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790512840836255-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790512840836255-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41279
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790512840836255-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41279
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:45:11.278721 21858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:45:11.279006 21858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:45:11.279048 21858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:45:11.282991 21858 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:45:11.283048 21858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:45:11.283067 21858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:45:11.283083 21858 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:45:11.283098 21858 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:45:11.288345 21858 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790512840836255-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790512840836255-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41279
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790512840836255-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790512840836255-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:45:11.289391 21858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:45:11.290464 21858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:45:11.298799 21864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:45:11.298820 21866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:45:11.298904 21863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:45:11.299420 21858 server_base.cc:1034] running on GCE node
I20260927 12:45:11.299944 21858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:45:11.301061 21858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:45:11.302250 21858 hybrid_clock.cc:648] HybridClock initialized: now 1790513111302238 us; error 28 us; skew 500 ppm
Sep 27 12:45:11 dist-test-slave-bnc1 krb5kdc[21845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:45:11.304855 21858 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 12:45:11.305800 21858 webserver.cc:460] Webserver started at http://127.0.0.1:43983/ using document root <none> and password file <none>
I20260927 12:45:11.306352 21858 fs_manager.cc:359] Metadata directory not provided
I20260927 12:45:11.306439 21858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:45:11.306666 21858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:45:11.308620 21858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790512840836255-392-0/minicluster-data/master-0/data/instance:
uuid: "682c19e7c62f4c83b33d2976ef066084"
format_stamp: "Formatted at 2026-09-27 12:45:11 on dist-test-slave-bnc1"
server_key: "1a23f1305506e06bf2c489637e4b0fe1"
server_key_iv: "68dc6d71b8b2fcd4dac40be940050bb4"
server_key_version: "encryptionkey@0"
I20260927 12:45:11.309176 21858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790512840836255-392-0/minicluster-data/master-0/wal/instance:
uuid: "682c19e7c62f4c83b33d2976ef066084"
format_stamp: "Formatted at 2026-09-27 12:45:11 on dist-test-slave-bnc1"
server_key: "1a23f1305506e06bf2c489637e4b0fe1"
server_key_iv: "68dc6d71b8b2fcd4dac40be940050bb4"
server_key_version: "encryptionkey@0"
I20260927 12:45:11.312736 21858 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260927 12:45:11.315151 21873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:45:11.316277 21858 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 12:45:11.316414 21858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790512840836255-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790512840836255-392-0/minicluster-data/master-0/wal
uuid: "682c19e7c62f4c83b33d2976ef066084"
format_stamp: "Formatted at 2026-09-27 12:45:11 on dist-test-slave-bnc1"
server_key: "1a23f1305506e06bf2c489637e4b0fe1"
server_key_iv: "68dc6d71b8b2fcd4dac40be940050bb4"
server_key_version: "encryptionkey@0"
I20260927 12:45:11.316524 21858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790512840836255-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790512840836255-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790512840836255-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:45:11.320936 21858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:45:11.323890 21858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:45:11.324083 21858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:45:11.330581 21858 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41279
I20260927 12:45:11.330648 21925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41279 every 8 connection(s)
I20260927 12:45:11.331477 21858 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:45:11.334297 21926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:45:11.336251 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 21858
I20260927 12:45:11.336391 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790512840836255-392-0/minicluster-data/master-0/wal/instance
I20260927 12:45:11.336687 392 external_mini_cluster.cc:1428] Setting key 3009db1a7f2cca41d8eea349546125cb
I20260927 12:45:11.341234 21926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 682c19e7c62f4c83b33d2976ef066084: Bootstrap starting.
I20260927 12:45:11.342007 21929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:11.338114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.119:53481 (local address 127.0.0.1:41279)
0927 12:45:11.338523 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:11.338542 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:11.338593 (+ 51us) server_negotiation.cc:407] Connection header received
0927 12:45:11.339228 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:11.339244 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:11.339519 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:11.339897 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:11.340687 (+ 790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:45:11.340692 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:45:11.340703 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:45:11.340713 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:45:11.340766 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:45:11.340785 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:45:11.340896 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 12:45:11.340997 (+ 101us) server_negotiation.cc:299] Negotiation successful
0927 12:45:11.341172 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":140,"threads_started":1}
I20260927 12:45:11.343819 21926 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 682c19e7c62f4c83b33d2976ef066084: Neither blocks nor log segments found. Creating new log.
I20260927 12:45:11.344627 21926 log.cc:824] T 00000000000000000000000000000000 P 682c19e7c62f4c83b33d2976ef066084: Log is configured to *not* fsync() on all Append() calls
I20260927 12:45:11.344914 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 12:45:11.346660 21926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 682c19e7c62f4c83b33d2976ef066084: No bootstrap required, opened a new log
I20260927 12:45:11.348636 21929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:11.347095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.119:56347 (local address 127.0.0.1:41279)
0927 12:45:11.347193 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:11.347197 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:11.347396 (+ 199us) server_negotiation.cc:407] Connection header received
0927 12:45:11.347471 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:11.347474 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:11.347504 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:11.347588 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:11.348268 (+ 680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:45:11.348271 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:45:11.348274 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:45:11.348284 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:45:11.348311 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:45:11.348325 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:45:11.348416 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 12:45:11.348493 (+ 77us) server_negotiation.cc:299] Negotiation successful
0927 12:45:11.348533 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260927 12:45:11.350543 21926 raft_consensus.cc:348] T 00000000000000000000000000000000 P 682c19e7c62f4c83b33d2976ef066084 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "682c19e7c62f4c83b33d2976ef066084" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41279 } }
I20260927 12:45:11.350741 21926 raft_consensus.cc:374] T 00000000000000000000000000000000 P 682c19e7c62f4c83b33d2976ef066084 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:45:11.350773 21926 raft_consensus.cc:729] T 00000000000000000000000000000000 P 682c19e7c62f4c83b33d2976ef066084 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 682c19e7c62f4c83b33d2976ef066084, State: Initialized, Role: FOLLOWER
I20260927 12:45:11.351150 21926 consensus_queue.cc:260] T 00000000000000000000000000000000 P 682c19e7c62f4c83b33d2976ef066084 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "682c19e7c62f4c83b33d2976ef066084" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41279 } }
I20260927 12:45:11.351264 21926 raft_consensus.cc:388] T 00000000000000000000000000000000 P 682c19e7c62f4c83b33d2976ef066084 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:45:11.351306 21926 raft_consensus.cc:482] T 00000000000000000000000000000000 P 682c19e7c62f4c83b33d2976ef066084 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:45:11.351374 21926 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 682c19e7c62f4c83b33d2976ef066084 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:45:11.352149 21926 raft_consensus.cc:504] T 00000000000000000000000000000000 P 682c19e7c62f4c83b33d2976ef066084 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "682c19e7c62f4c83b33d2976ef066084" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41279 } }
I20260927 12:45:11.352418 21926 leader_election.cc:302] T 00000000000000000000000000000000 P 682c19e7c62f4c83b33d2976ef066084 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 682c19e7c62f4c83b33d2976ef066084; no voters:
I20260927 12:45:11.352648 21926 leader_election.cc:288] T 00000000000000000000000000000000 P 682c19e7c62f4c83b33d2976ef066084 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:45:11.352783 21936 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 682c19e7c62f4c83b33d2976ef066084 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:45:11.353000 21936 raft_consensus.cc:686] T 00000000000000000000000000000000 P 682c19e7c62f4c83b33d2976ef066084 [term 1 LEADER]: Becoming Leader. State: Replica: 682c19e7c62f4c83b33d2976ef066084, State: Running, Role: LEADER
I20260927 12:45:11.353499 21936 consensus_queue.cc:237] T 00000000000000000000000000000000 P 682c19e7c62f4c83b33d2976ef066084 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "682c19e7c62f4c83b33d2976ef066084" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41279 } }
I20260927 12:45:11.353906 21926 sys_catalog.cc:564] T 00000000000000000000000000000000 P 682c19e7c62f4c83b33d2976ef066084 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:45:11.355167 21938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 682c19e7c62f4c83b33d2976ef066084 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 682c19e7c62f4c83b33d2976ef066084. Latest consensus state: current_term: 1 leader_uuid: "682c19e7c62f4c83b33d2976ef066084" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "682c19e7c62f4c83b33d2976ef066084" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41279 } } }
I20260927 12:45:11.355283 21938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 682c19e7c62f4c83b33d2976ef066084 [sys.catalog]: This master's current role is: LEADER
I20260927 12:45:11.355648 21946 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:45:11.355630 21937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 682c19e7c62f4c83b33d2976ef066084 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "682c19e7c62f4c83b33d2976ef066084" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "682c19e7c62f4c83b33d2976ef066084" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41279 } } }
I20260927 12:45:11.355770 21937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 682c19e7c62f4c83b33d2976ef066084 [sys.catalog]: This master's current role is: LEADER
I20260927 12:45:11.358626 21946 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:45:11.363891 21946 catalog_manager.cc:1347] Generated new cluster ID: 9d971503f17f47e7abd1a77aa71e5c77
I20260927 12:45:11.363962 21946 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:45:11.384531 21946 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:45:11.384920 21946 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:45:11.392050 21946 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 682c19e7c62f4c83b33d2976ef066084: Generated new TSK 0
I20260927 12:45:11.392722 21946 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:45:11.401539 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 21858
2026-09-27T12:45:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1373 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790512840836255-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:45:11 dist-test-slave-bnc1 krb5kdc[21957](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:45:11 dist-test-slave-bnc1 krb5kdc[21957](info): set up 2 sockets
Sep 27 12:45:11 dist-test-slave-bnc1 krb5kdc[21957](info): commencing operation
krb5kdc: starting...
W20260927 12:45:12.449776 392 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790512840836255-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790512840836255-392-0/krb5kdc/test-user.keytab.
Sep 27 12:45:12 dist-test-slave-bnc1 krb5kdc[21957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:45:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:45:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:45:12.567883 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790512840836255-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790512840836255-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790512840836255-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45971
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45971
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:45:12.679740 21970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:45:12.680002 21970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:45:12.680044 21970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:45:12.684088 21970 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:45:12.684144 21970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:45:12.684163 21970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:45:12.684180 21970 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:45:12.684195 21970 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:45:12.689405 21970 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790512840836255-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790512840836255-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45971
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790512840836255-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:45:12.690450 21970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:45:12.691465 21970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:45:12.700014 21975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:45:12.700014 21978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:45:12.700033 21976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:45:12.700273 21970 server_base.cc:1034] running on GCE node
I20260927 12:45:12.700831 21970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:45:12.702015 21970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:45:12.703222 21970 hybrid_clock.cc:648] HybridClock initialized: now 1790513112703200 us; error 33 us; skew 500 ppm
Sep 27 12:45:12 dist-test-slave-bnc1 krb5kdc[21957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513112, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:45:12.706157 21970 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:45:12.706983 21970 webserver.cc:460] Webserver started at http://127.0.98.62:44883/ using document root <none> and password file <none>
I20260927 12:45:12.707497 21970 fs_manager.cc:359] Metadata directory not provided
I20260927 12:45:12.707556 21970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:45:12.707724 21970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:45:12.709743 21970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790512840836255-392-0/minicluster-data/master-0/data/instance:
uuid: "207e490c0c0249d6a5bc3cb55b3ab1e4"
format_stamp: "Formatted at 2026-09-27 12:45:12 on dist-test-slave-bnc1"
server_key: "cf21a81e171ea96a1aabf7af78021f67"
server_key_iv: "b2bdbe47bdcd6a50bb89e8d033e814dc"
server_key_version: "encryptionkey@0"
I20260927 12:45:12.710239 21970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790512840836255-392-0/minicluster-data/master-0/wal/instance:
uuid: "207e490c0c0249d6a5bc3cb55b3ab1e4"
format_stamp: "Formatted at 2026-09-27 12:45:12 on dist-test-slave-bnc1"
server_key: "cf21a81e171ea96a1aabf7af78021f67"
server_key_iv: "b2bdbe47bdcd6a50bb89e8d033e814dc"
server_key_version: "encryptionkey@0"
I20260927 12:45:12.713735 21970 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:45:12.716086 21985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:45:12.717252 21970 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:45:12.717348 21970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790512840836255-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790512840836255-392-0/minicluster-data/master-0/wal
uuid: "207e490c0c0249d6a5bc3cb55b3ab1e4"
format_stamp: "Formatted at 2026-09-27 12:45:12 on dist-test-slave-bnc1"
server_key: "cf21a81e171ea96a1aabf7af78021f67"
server_key_iv: "b2bdbe47bdcd6a50bb89e8d033e814dc"
server_key_version: "encryptionkey@0"
I20260927 12:45:12.717432 21970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790512840836255-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790512840836255-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790512840836255-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:45:12.742415 21970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:45:12.745847 21970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:45:12.745998 21970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:45:12.752714 21970 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45971
I20260927 12:45:12.752741 22037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45971 every 8 connection(s)
I20260927 12:45:12.753669 21970 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:45:12.754053 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 21970
I20260927 12:45:12.754207 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790512840836255-392-0/minicluster-data/master-0/wal/instance
I20260927 12:45:12.754477 392 external_mini_cluster.cc:1428] Setting key e50b82343d3483403081dd855228354d
I20260927 12:45:12.756744 22038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:45:12.762135 22038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 207e490c0c0249d6a5bc3cb55b3ab1e4: Bootstrap starting.
Sep 27 12:45:12 dist-test-slave-bnc1 krb5kdc[21957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:45:12.764430 22038 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 207e490c0c0249d6a5bc3cb55b3ab1e4: Neither blocks nor log segments found. Creating new log.
I20260927 12:45:12.765257 22038 log.cc:824] T 00000000000000000000000000000000 P 207e490c0c0249d6a5bc3cb55b3ab1e4: Log is configured to *not* fsync() on all Append() calls
I20260927 12:45:12.767465 22038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 207e490c0c0249d6a5bc3cb55b3ab1e4: No bootstrap required, opened a new log
I20260927 12:45:12.768600 22041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:12.755852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58971 (local address 127.0.98.62:45971)
0927 12:45:12.756296 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:12.756309 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:12.756343 (+ 34us) server_negotiation.cc:407] Connection header received
0927 12:45:12.757134 (+ 791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:12.757151 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:12.757650 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:12.757976 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:12.758764 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:12.759738 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:12.760613 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:12.761089 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:12.761258 (+ 169us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:45:12.763822 (+ 2564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:45:12.763826 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:45:12.763838 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:45:12.763865 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:45:12.765698 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:45:12.766296 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:45:12.766301 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:45:12.766305 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:45:12.766420 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:45:12.766749 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:45:12.766752 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:45:12.766753 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:45:12.767107 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:45:12.767286 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 12:45:12.767636 (+ 350us) server_negotiation.cc:299] Negotiation successful
0927 12:45:12.767897 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":140,"threads_started":1}
I20260927 12:45:12.771595 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 12:45:12.771415 22038 raft_consensus.cc:348] T 00000000000000000000000000000000 P 207e490c0c0249d6a5bc3cb55b3ab1e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "207e490c0c0249d6a5bc3cb55b3ab1e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45971 } }
I20260927 12:45:12.771663 22038 raft_consensus.cc:374] T 00000000000000000000000000000000 P 207e490c0c0249d6a5bc3cb55b3ab1e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:45:12.771719 22038 raft_consensus.cc:729] T 00000000000000000000000000000000 P 207e490c0c0249d6a5bc3cb55b3ab1e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 207e490c0c0249d6a5bc3cb55b3ab1e4, State: Initialized, Role: FOLLOWER
I20260927 12:45:12.772181 22038 consensus_queue.cc:260] T 00000000000000000000000000000000 P 207e490c0c0249d6a5bc3cb55b3ab1e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "207e490c0c0249d6a5bc3cb55b3ab1e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45971 } }
I20260927 12:45:12.772334 22038 raft_consensus.cc:388] T 00000000000000000000000000000000 P 207e490c0c0249d6a5bc3cb55b3ab1e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:45:12.772413 22038 raft_consensus.cc:482] T 00000000000000000000000000000000 P 207e490c0c0249d6a5bc3cb55b3ab1e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:45:12.772514 22038 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 207e490c0c0249d6a5bc3cb55b3ab1e4 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:45:12.773718 22038 raft_consensus.cc:504] T 00000000000000000000000000000000 P 207e490c0c0249d6a5bc3cb55b3ab1e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "207e490c0c0249d6a5bc3cb55b3ab1e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45971 } }
I20260927 12:45:12.774088 22038 leader_election.cc:302] T 00000000000000000000000000000000 P 207e490c0c0249d6a5bc3cb55b3ab1e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 207e490c0c0249d6a5bc3cb55b3ab1e4; no voters:
I20260927 12:45:12.774422 22038 leader_election.cc:288] T 00000000000000000000000000000000 P 207e490c0c0249d6a5bc3cb55b3ab1e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:45:12.774535 22048 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 207e490c0c0249d6a5bc3cb55b3ab1e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:45:12.774744 22048 raft_consensus.cc:686] T 00000000000000000000000000000000 P 207e490c0c0249d6a5bc3cb55b3ab1e4 [term 1 LEADER]: Becoming Leader. State: Replica: 207e490c0c0249d6a5bc3cb55b3ab1e4, State: Running, Role: LEADER
I20260927 12:45:12.775027 22048 consensus_queue.cc:237] T 00000000000000000000000000000000 P 207e490c0c0249d6a5bc3cb55b3ab1e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "207e490c0c0249d6a5bc3cb55b3ab1e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45971 } }
I20260927 12:45:12.775669 22038 sys_catalog.cc:564] T 00000000000000000000000000000000 P 207e490c0c0249d6a5bc3cb55b3ab1e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:45:12.776190 22048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 207e490c0c0249d6a5bc3cb55b3ab1e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 207e490c0c0249d6a5bc3cb55b3ab1e4. Latest consensus state: current_term: 1 leader_uuid: "207e490c0c0249d6a5bc3cb55b3ab1e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "207e490c0c0249d6a5bc3cb55b3ab1e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45971 } } }
I20260927 12:45:12.776315 22048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 207e490c0c0249d6a5bc3cb55b3ab1e4 [sys.catalog]: This master's current role is: LEADER
I20260927 12:45:12.778277 22054 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:45:12.781333 22054 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:45:12.783020 22041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:12.773770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48623 (local address 127.0.98.62:45971)
0927 12:45:12.773896 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:12.773900 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:12.774074 (+ 174us) server_negotiation.cc:407] Connection header received
0927 12:45:12.774193 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:12.774195 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:12.774269 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:12.774331 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:12.775113 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:12.775874 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:12.777145 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:12.777472 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:12.777604 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:45:12.779332 (+ 1728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:45:12.779338 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:45:12.779340 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:45:12.779366 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:45:12.781043 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:45:12.781629 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:45:12.781632 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:45:12.781634 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:45:12.781720 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:45:12.782099 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:45:12.782103 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:45:12.782104 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:45:12.782319 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:45:12.782472 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 12:45:12.782766 (+ 294us) server_negotiation.cc:299] Negotiation successful
0927 12:45:12.782866 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 12:45:12.778410 22049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 207e490c0c0249d6a5bc3cb55b3ab1e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "207e490c0c0249d6a5bc3cb55b3ab1e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "207e490c0c0249d6a5bc3cb55b3ab1e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45971 } } }
I20260927 12:45:12.783354 22049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 207e490c0c0249d6a5bc3cb55b3ab1e4 [sys.catalog]: This master's current role is: LEADER
I20260927 12:45:12.786744 22054 catalog_manager.cc:1347] Generated new cluster ID: 96e7b2f3dfae43ee93e4a18725756347
I20260927 12:45:12.786804 22054 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:45:12.828888 22054 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:45:12.829957 22054 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:45:12.835939 22054 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 207e490c0c0249d6a5bc3cb55b3ab1e4: Generated new TSK 0
I20260927 12:45:12.836777 22054 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:45:12.852064 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 21970
W20260927 12:45:12.857205 21955 connection.cc:535] client connection to 127.0.98.62:45971 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:45971): Connection reset by peer (error 104)
2026-09-27T12:45:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1450 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790512840836255-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:45:12 dist-test-slave-bnc1 krb5kdc[22069](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:45:12 dist-test-slave-bnc1 krb5kdc[22069](info): set up 2 sockets
Sep 27 12:45:12 dist-test-slave-bnc1 krb5kdc[22069](info): commencing operation
krb5kdc: starting...
W20260927 12:45:13.901926 392 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790512840836255-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790512840836255-392-0/krb5kdc/test-user.keytab.
Sep 27 12:45:13 dist-test-slave-bnc1 krb5kdc[22069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513113, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:45:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:45:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:45:14.022635 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790512840836255-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790512840836255-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790512840836255-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34851
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34851
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:45:14.131916 22082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:45:14.132164 22082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:45:14.132203 22082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:45:14.136170 22082 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:45:14.136237 22082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:45:14.136258 22082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:45:14.136274 22082 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:45:14.136289 22082 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:45:14.141356 22082 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790512840836255-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790512840836255-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34851
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22082
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790512840836255-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:45:14.142386 22082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:45:14.143342 22082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:45:14.151836 22088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:45:14.151909 22087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:45:14.151865 22090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:45:14.151990 22082 server_base.cc:1034] running on GCE node
I20260927 12:45:14.152624 22082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:45:14.153751 22082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:45:14.154934 22082 hybrid_clock.cc:648] HybridClock initialized: now 1790513114154922 us; error 27 us; skew 500 ppm
Sep 27 12:45:14 dist-test-slave-bnc1 krb5kdc[22069](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513114, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:45:14.162751 22082 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:45:14.163683 22082 webserver.cc:460] Webserver started at http://127.0.98.62:46815/ using document root <none> and password file <none>
I20260927 12:45:14.164240 22082 fs_manager.cc:359] Metadata directory not provided
I20260927 12:45:14.164330 22082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:45:14.164563 22082 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:45:14.166684 22082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790512840836255-392-0/minicluster-data/master-0/data/instance:
uuid: "9a2af4dfbe3e441ba2b5c845a733912a"
format_stamp: "Formatted at 2026-09-27 12:45:14 on dist-test-slave-bnc1"
server_key: "cf6caf28c2c6729a1c5d002a102b5ded"
server_key_iv: "6bb32a2170ce1f408e814adf34c5437a"
server_key_version: "encryptionkey@0"
I20260927 12:45:14.167243 22082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790512840836255-392-0/minicluster-data/master-0/wal/instance:
uuid: "9a2af4dfbe3e441ba2b5c845a733912a"
format_stamp: "Formatted at 2026-09-27 12:45:14 on dist-test-slave-bnc1"
server_key: "cf6caf28c2c6729a1c5d002a102b5ded"
server_key_iv: "6bb32a2170ce1f408e814adf34c5437a"
server_key_version: "encryptionkey@0"
I20260927 12:45:14.170894 22082 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 12:45:14.173269 22097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:45:14.174503 22082 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 12:45:14.174661 22082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790512840836255-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790512840836255-392-0/minicluster-data/master-0/wal
uuid: "9a2af4dfbe3e441ba2b5c845a733912a"
format_stamp: "Formatted at 2026-09-27 12:45:14 on dist-test-slave-bnc1"
server_key: "cf6caf28c2c6729a1c5d002a102b5ded"
server_key_iv: "6bb32a2170ce1f408e814adf34c5437a"
server_key_version: "encryptionkey@0"
I20260927 12:45:14.174774 22082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790512840836255-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790512840836255-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790512840836255-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:45:14.191708 22082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:45:14.194644 22082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:45:14.194844 22082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:45:14.201773 22082 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34851
I20260927 12:45:14.201797 22149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34851 every 8 connection(s)
I20260927 12:45:14.202674 22082 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:45:14.205247 22150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:45:14.209062 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 22082
I20260927 12:45:14.209200 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790512840836255-392-0/minicluster-data/master-0/wal/instance
I20260927 12:45:14.209482 392 external_mini_cluster.cc:1428] Setting key e5468502e8ec58b036772a003a0177c7
I20260927 12:45:14.210714 22150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a2af4dfbe3e441ba2b5c845a733912a: Bootstrap starting.
I20260927 12:45:14.212904 22150 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a2af4dfbe3e441ba2b5c845a733912a: Neither blocks nor log segments found. Creating new log.
I20260927 12:45:14.213683 22150 log.cc:824] T 00000000000000000000000000000000 P 9a2af4dfbe3e441ba2b5c845a733912a: Log is configured to *not* fsync() on all Append() calls
I20260927 12:45:14.215911 22150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a2af4dfbe3e441ba2b5c845a733912a: No bootstrap required, opened a new log
I20260927 12:45:14.218839 22153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:14.210872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37715 (local address 127.0.98.62:34851)
0927 12:45:14.211280 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:14.211294 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:14.211329 (+ 35us) server_negotiation.cc:407] Connection header received
0927 12:45:14.211945 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:14.211960 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:14.212295 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:14.212625 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:14.213051 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:14.214295 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:14.215583 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:14.216099 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:14.216216 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:45:14.217130 (+ 914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:45:14.217134 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:45:14.217147 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:45:14.217520 (+ 373us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:45:14.217630 (+ 110us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:45:14.217668 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:45:14.217769 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 12:45:14.217971 (+ 202us) server_negotiation.cc:299] Negotiation successful
0927 12:45:14.218141 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":135,"threads_started":1}
I20260927 12:45:14.219877 22150 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a2af4dfbe3e441ba2b5c845a733912a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a2af4dfbe3e441ba2b5c845a733912a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34851 } }
I20260927 12:45:14.220088 22150 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a2af4dfbe3e441ba2b5c845a733912a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:45:14.220156 22150 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a2af4dfbe3e441ba2b5c845a733912a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a2af4dfbe3e441ba2b5c845a733912a, State: Initialized, Role: FOLLOWER
I20260927 12:45:14.220592 22150 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a2af4dfbe3e441ba2b5c845a733912a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a2af4dfbe3e441ba2b5c845a733912a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34851 } }
I20260927 12:45:14.220763 22150 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a2af4dfbe3e441ba2b5c845a733912a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:45:14.220862 22150 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a2af4dfbe3e441ba2b5c845a733912a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:45:14.220983 22150 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a2af4dfbe3e441ba2b5c845a733912a [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:45:14.221745 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 12:45:14.222025 22150 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a2af4dfbe3e441ba2b5c845a733912a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a2af4dfbe3e441ba2b5c845a733912a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34851 } }
I20260927 12:45:14.222386 22150 leader_election.cc:302] T 00000000000000000000000000000000 P 9a2af4dfbe3e441ba2b5c845a733912a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a2af4dfbe3e441ba2b5c845a733912a; no voters:
I20260927 12:45:14.222735 22150 leader_election.cc:288] T 00000000000000000000000000000000 P 9a2af4dfbe3e441ba2b5c845a733912a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:45:14.222822 22155 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a2af4dfbe3e441ba2b5c845a733912a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:45:14.223099 22155 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a2af4dfbe3e441ba2b5c845a733912a [term 1 LEADER]: Becoming Leader. State: Replica: 9a2af4dfbe3e441ba2b5c845a733912a, State: Running, Role: LEADER
I20260927 12:45:14.223440 22155 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a2af4dfbe3e441ba2b5c845a733912a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a2af4dfbe3e441ba2b5c845a733912a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34851 } }
I20260927 12:45:14.223968 22150 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a2af4dfbe3e441ba2b5c845a733912a [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:45:14.225131 22156 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a2af4dfbe3e441ba2b5c845a733912a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a2af4dfbe3e441ba2b5c845a733912a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a2af4dfbe3e441ba2b5c845a733912a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34851 } } }
I20260927 12:45:14.225245 22156 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a2af4dfbe3e441ba2b5c845a733912a [sys.catalog]: This master's current role is: LEADER
I20260927 12:45:14.225627 22159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a2af4dfbe3e441ba2b5c845a733912a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a2af4dfbe3e441ba2b5c845a733912a. Latest consensus state: current_term: 1 leader_uuid: "9a2af4dfbe3e441ba2b5c845a733912a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a2af4dfbe3e441ba2b5c845a733912a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34851 } } }
I20260927 12:45:14.226550 22159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a2af4dfbe3e441ba2b5c845a733912a [sys.catalog]: This master's current role is: LEADER
W20260927 12:45:14.228971 22175 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9a2af4dfbe3e441ba2b5c845a733912a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 12:45:14.229054 22175 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 12:45:14.229239 22169 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:45:14.229974 22153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:14.224888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59403 (local address 127.0.98.62:34851)
0927 12:45:14.224989 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:14.224993 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:14.225520 (+ 527us) server_negotiation.cc:407] Connection header received
0927 12:45:14.225726 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:14.225729 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:14.225774 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:14.225825 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:14.226288 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:14.226999 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:14.228102 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:14.228475 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:14.228593 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:45:14.229464 (+ 871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:45:14.229467 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:45:14.229470 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:45:14.229482 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:45:14.229513 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:45:14.229525 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:45:14.229675 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 12:45:14.229800 (+ 125us) server_negotiation.cc:299] Negotiation successful
0927 12:45:14.229830 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 12:45:14.230144 22169 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:45:14.235520 22169 catalog_manager.cc:1347] Generated new cluster ID: b3316ec46b09474e9bae86a3f300e42f
I20260927 12:45:14.235597 22169 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:45:14.257025 22169 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:45:14.258065 22169 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:45:14.263773 22169 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a2af4dfbe3e441ba2b5c845a733912a: Generated new TSK 0
I20260927 12:45:14.264559 22169 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:45:14.277237 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 22082
W20260927 12:45:14.282049 22067 connection.cc:535] client connection to 127.0.98.62:34851 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:34851): Connection reset by peer (error 104)
2026-09-27T12:45:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1424 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790512840836255-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:45:14 dist-test-slave-bnc1 krb5kdc[22181](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:45:14 dist-test-slave-bnc1 krb5kdc[22181](info): set up 2 sockets
Sep 27 12:45:14 dist-test-slave-bnc1 krb5kdc[22181](info): commencing operation
krb5kdc: starting...
W20260927 12:45:15.309831 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790512840836255-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790512840836255-392-0/krb5kdc/test-user.keytab.
Sep 27 12:45:15 dist-test-slave-bnc1 krb5kdc[22181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:45:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:45:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:45:15.430262 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790512840836255-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790512840836255-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790512840836255-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44821
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44821
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:45:15.539985 22194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:45:15.540249 22194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:45:15.540292 22194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:45:15.544806 22194 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:45:15.544863 22194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:45:15.544883 22194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:45:15.544899 22194 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:45:15.544914 22194 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:45:15.550200 22194 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790512840836255-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790512840836255-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44821
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790512840836255-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:45:15.551189 22194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:45:15.552141 22194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:45:15.560639 22200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:45:15.560858 22199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:45:15.560876 22194 server_base.cc:1034] running on GCE node
W20260927 12:45:15.560873 22202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:45:15.561636 22194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:45:15.562737 22194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:45:15.563930 22194 hybrid_clock.cc:648] HybridClock initialized: now 1790513115563919 us; error 31 us; skew 500 ppm
Sep 27 12:45:15 dist-test-slave-bnc1 krb5kdc[22181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513115, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:45:15.566687 22194 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:45:15.567482 22194 webserver.cc:460] Webserver started at http://127.0.98.62:35349/ using document root <none> and password file <none>
I20260927 12:45:15.567991 22194 fs_manager.cc:359] Metadata directory not provided
I20260927 12:45:15.568048 22194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:45:15.568203 22194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:45:15.570232 22194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790512840836255-392-0/minicluster-data/master-0/data/instance:
uuid: "181eea346f074ea299c44d5fb483e5c1"
format_stamp: "Formatted at 2026-09-27 12:45:15 on dist-test-slave-bnc1"
server_key: "ac21d01ad8eeed5aff9b2ce724f21f04"
server_key_iv: "8c8d7aeab885d37b980eb153fcebdc3a"
server_key_version: "encryptionkey@0"
I20260927 12:45:15.570739 22194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790512840836255-392-0/minicluster-data/master-0/wal/instance:
uuid: "181eea346f074ea299c44d5fb483e5c1"
format_stamp: "Formatted at 2026-09-27 12:45:15 on dist-test-slave-bnc1"
server_key: "ac21d01ad8eeed5aff9b2ce724f21f04"
server_key_iv: "8c8d7aeab885d37b980eb153fcebdc3a"
server_key_version: "encryptionkey@0"
I20260927 12:45:15.574266 22194 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 12:45:15.576529 22209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:45:15.577759 22194 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 12:45:15.577881 22194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790512840836255-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790512840836255-392-0/minicluster-data/master-0/wal
uuid: "181eea346f074ea299c44d5fb483e5c1"
format_stamp: "Formatted at 2026-09-27 12:45:15 on dist-test-slave-bnc1"
server_key: "ac21d01ad8eeed5aff9b2ce724f21f04"
server_key_iv: "8c8d7aeab885d37b980eb153fcebdc3a"
server_key_version: "encryptionkey@0"
I20260927 12:45:15.577979 22194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790512840836255-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790512840836255-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790512840836255-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:45:15.582451 22194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:45:15.585316 22194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:45:15.585530 22194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:45:15.592491 22194 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44821
I20260927 12:45:15.592517 22261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44821 every 8 connection(s)
I20260927 12:45:15.593443 22194 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:45:15.596174 22262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:45:15.596400 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 22194
I20260927 12:45:15.596531 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790512840836255-392-0/minicluster-data/master-0/wal/instance
I20260927 12:45:15.596848 392 external_mini_cluster.cc:1428] Setting key 860bfa30f2c4c770d5b106cd0ed8352e
I20260927 12:45:15.602018 22265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:15.598248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57309 (local address 127.0.98.62:44821)
0927 12:45:15.598653 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:15.598667 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:15.598702 (+ 35us) server_negotiation.cc:407] Connection header received
0927 12:45:15.599466 (+ 764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:15.599483 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:15.599751 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:15.600088 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:15.600826 (+ 738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:45:15.600831 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:45:15.600853 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:45:15.600864 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:45:15.600916 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:45:15.600936 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:45:15.601083 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 12:45:15.601153 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 12:45:15.601294 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":120,"threads_started":1}
I20260927 12:45:15.602787 22262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 181eea346f074ea299c44d5fb483e5c1: Bootstrap starting.
I20260927 12:45:15.604640 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 12:45:15.605090 22262 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 181eea346f074ea299c44d5fb483e5c1: Neither blocks nor log segments found. Creating new log.
I20260927 12:45:15.605885 22262 log.cc:824] T 00000000000000000000000000000000 P 181eea346f074ea299c44d5fb483e5c1: Log is configured to *not* fsync() on all Append() calls
I20260927 12:45:15.608114 22262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 181eea346f074ea299c44d5fb483e5c1: No bootstrap required, opened a new log
I20260927 12:45:15.608343 22265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:15.606680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34831 (local address 127.0.98.62:44821)
0927 12:45:15.606818 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:15.606822 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:15.607030 (+ 208us) server_negotiation.cc:407] Connection header received
0927 12:45:15.607156 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:15.607158 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:15.607184 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:15.607245 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:15.607928 (+ 683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:45:15.607930 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:45:15.607932 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:45:15.607941 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:45:15.607964 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:45:15.607974 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:45:15.608041 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 12:45:15.608157 (+ 116us) server_negotiation.cc:299] Negotiation successful
0927 12:45:15.608217 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 12:45:15.611894 22262 raft_consensus.cc:348] T 00000000000000000000000000000000 P 181eea346f074ea299c44d5fb483e5c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "181eea346f074ea299c44d5fb483e5c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44821 } }
I20260927 12:45:15.612141 22262 raft_consensus.cc:374] T 00000000000000000000000000000000 P 181eea346f074ea299c44d5fb483e5c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:45:15.612222 22262 raft_consensus.cc:729] T 00000000000000000000000000000000 P 181eea346f074ea299c44d5fb483e5c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 181eea346f074ea299c44d5fb483e5c1, State: Initialized, Role: FOLLOWER
I20260927 12:45:15.612668 22262 consensus_queue.cc:260] T 00000000000000000000000000000000 P 181eea346f074ea299c44d5fb483e5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "181eea346f074ea299c44d5fb483e5c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44821 } }
I20260927 12:45:15.612829 22262 raft_consensus.cc:388] T 00000000000000000000000000000000 P 181eea346f074ea299c44d5fb483e5c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:45:15.612900 22262 raft_consensus.cc:482] T 00000000000000000000000000000000 P 181eea346f074ea299c44d5fb483e5c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:45:15.613013 22262 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 181eea346f074ea299c44d5fb483e5c1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:45:15.614095 22262 raft_consensus.cc:504] T 00000000000000000000000000000000 P 181eea346f074ea299c44d5fb483e5c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "181eea346f074ea299c44d5fb483e5c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44821 } }
I20260927 12:45:15.614454 22262 leader_election.cc:302] T 00000000000000000000000000000000 P 181eea346f074ea299c44d5fb483e5c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 181eea346f074ea299c44d5fb483e5c1; no voters:
I20260927 12:45:15.614807 22262 leader_election.cc:288] T 00000000000000000000000000000000 P 181eea346f074ea299c44d5fb483e5c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:45:15.614888 22272 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 181eea346f074ea299c44d5fb483e5c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:45:15.615150 22272 raft_consensus.cc:686] T 00000000000000000000000000000000 P 181eea346f074ea299c44d5fb483e5c1 [term 1 LEADER]: Becoming Leader. State: Replica: 181eea346f074ea299c44d5fb483e5c1, State: Running, Role: LEADER
I20260927 12:45:15.615528 22272 consensus_queue.cc:237] T 00000000000000000000000000000000 P 181eea346f074ea299c44d5fb483e5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "181eea346f074ea299c44d5fb483e5c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44821 } }
I20260927 12:45:15.616122 22262 sys_catalog.cc:564] T 00000000000000000000000000000000 P 181eea346f074ea299c44d5fb483e5c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:45:15.617255 22273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 181eea346f074ea299c44d5fb483e5c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "181eea346f074ea299c44d5fb483e5c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "181eea346f074ea299c44d5fb483e5c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44821 } } }
I20260927 12:45:15.617249 22274 sys_catalog.cc:455] T 00000000000000000000000000000000 P 181eea346f074ea299c44d5fb483e5c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 181eea346f074ea299c44d5fb483e5c1. Latest consensus state: current_term: 1 leader_uuid: "181eea346f074ea299c44d5fb483e5c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "181eea346f074ea299c44d5fb483e5c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44821 } } }
I20260927 12:45:15.617389 22273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 181eea346f074ea299c44d5fb483e5c1 [sys.catalog]: This master's current role is: LEADER
I20260927 12:45:15.617389 22274 sys_catalog.cc:458] T 00000000000000000000000000000000 P 181eea346f074ea299c44d5fb483e5c1 [sys.catalog]: This master's current role is: LEADER
I20260927 12:45:15.618538 22281 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:45:15.621747 22281 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:45:15.626829 22281 catalog_manager.cc:1347] Generated new cluster ID: ee729de4c61247c794b7cbeaa56c44f6
I20260927 12:45:15.626905 22281 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:45:15.639539 22281 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:45:15.639874 22281 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:45:15.644235 22281 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 181eea346f074ea299c44d5fb483e5c1: Generated new TSK 0
I20260927 12:45:15.644918 22281 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:45:15.652124 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 22194
2026-09-27T12:45:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1375 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790512840836255-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:45:15 dist-test-slave-bnc1 krb5kdc[22293](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:45:15 dist-test-slave-bnc1 krb5kdc[22293](info): set up 2 sockets
Sep 27 12:45:15 dist-test-slave-bnc1 krb5kdc[22293](info): commencing operation
krb5kdc: starting...
W20260927 12:45:16.689883 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790512840836255-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790512840836255-392-0/krb5kdc/test-user.keytab.
Sep 27 12:45:16 dist-test-slave-bnc1 krb5kdc[22293](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:45:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:45:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:45:16.808801 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790512840836255-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790512840836255-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790512840836255-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35353
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35353
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:45:16.921044 22306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:45:16.921334 22306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:45:16.921419 22306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:45:16.925609 22306 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:45:16.925680 22306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:45:16.925714 22306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:45:16.925751 22306 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:45:16.925784 22306 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:45:16.930999 22306 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790512840836255-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790512840836255-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35353
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790512840836255-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:45:16.932081 22306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:45:16.933096 22306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:45:16.941649 22311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:45:16.941653 22314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:45:16.941649 22312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:45:16.941810 22306 server_base.cc:1034] running on GCE node
I20260927 12:45:16.942425 22306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:45:16.943472 22306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:45:16.944646 22306 hybrid_clock.cc:648] HybridClock initialized: now 1790513116944629 us; error 30 us; skew 500 ppm
Sep 27 12:45:16 dist-test-slave-bnc1 krb5kdc[22293](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513116, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:45:16.947443 22306 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:45:16.948304 22306 webserver.cc:460] Webserver started at http://127.0.98.62:43197/ using document root <none> and password file <none>
I20260927 12:45:16.948863 22306 fs_manager.cc:359] Metadata directory not provided
I20260927 12:45:16.948956 22306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:45:16.949187 22306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:45:16.951260 22306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790512840836255-392-0/minicluster-data/master-0/data/instance:
uuid: "b7919880ff1c45e08b99766137495c23"
format_stamp: "Formatted at 2026-09-27 12:45:16 on dist-test-slave-bnc1"
server_key: "bbe7688b3abb5224c7b83302eae886d9"
server_key_iv: "ee101d531e0a5cdc14ef933d99f81558"
server_key_version: "encryptionkey@0"
I20260927 12:45:16.951826 22306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790512840836255-392-0/minicluster-data/master-0/wal/instance:
uuid: "b7919880ff1c45e08b99766137495c23"
format_stamp: "Formatted at 2026-09-27 12:45:16 on dist-test-slave-bnc1"
server_key: "bbe7688b3abb5224c7b83302eae886d9"
server_key_iv: "ee101d531e0a5cdc14ef933d99f81558"
server_key_version: "encryptionkey@0"
I20260927 12:45:16.955448 22306 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260927 12:45:16.957878 22321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:45:16.959079 22306 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 12:45:16.959215 22306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790512840836255-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790512840836255-392-0/minicluster-data/master-0/wal
uuid: "b7919880ff1c45e08b99766137495c23"
format_stamp: "Formatted at 2026-09-27 12:45:16 on dist-test-slave-bnc1"
server_key: "bbe7688b3abb5224c7b83302eae886d9"
server_key_iv: "ee101d531e0a5cdc14ef933d99f81558"
server_key_version: "encryptionkey@0"
I20260927 12:45:16.959324 22306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790512840836255-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790512840836255-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790512840836255-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:45:16.984488 22306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:45:16.987537 22306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:45:16.987751 22306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:45:16.994673 22306 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35353
I20260927 12:45:16.994695 22373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35353 every 8 connection(s)
I20260927 12:45:16.995666 22306 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:45:16.998387 22374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:45:17.003731 22374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7919880ff1c45e08b99766137495c23: Bootstrap starting.
I20260927 12:45:17.005023 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 22306
I20260927 12:45:17.005170 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790512840836255-392-0/minicluster-data/master-0/wal/instance
I20260927 12:45:17.005445 392 external_mini_cluster.cc:1428] Setting key 91cd42a11091780eed921928c0c2acf3
I20260927 12:45:17.006484 22374 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7919880ff1c45e08b99766137495c23: Neither blocks nor log segments found. Creating new log.
I20260927 12:45:17.007280 22374 log.cc:824] T 00000000000000000000000000000000 P b7919880ff1c45e08b99766137495c23: Log is configured to *not* fsync() on all Append() calls
I20260927 12:45:17.009099 22374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7919880ff1c45e08b99766137495c23: No bootstrap required, opened a new log
I20260927 12:45:17.013419 22374 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7919880ff1c45e08b99766137495c23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7919880ff1c45e08b99766137495c23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35353 } }
I20260927 12:45:17.013727 22374 raft_consensus.cc:374] T 00000000000000000000000000000000 P b7919880ff1c45e08b99766137495c23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:45:17.013810 22374 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7919880ff1c45e08b99766137495c23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7919880ff1c45e08b99766137495c23, State: Initialized, Role: FOLLOWER
I20260927 12:45:17.014349 22374 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7919880ff1c45e08b99766137495c23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7919880ff1c45e08b99766137495c23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35353 } }
I20260927 12:45:17.014508 22374 raft_consensus.cc:388] T 00000000000000000000000000000000 P b7919880ff1c45e08b99766137495c23 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:45:17.014585 22374 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7919880ff1c45e08b99766137495c23 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:45:17.014688 22374 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7919880ff1c45e08b99766137495c23 [term 0 FOLLOWER]: Advancing to term 1
Sep 27 12:45:17 dist-test-slave-bnc1 krb5kdc[22293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:45:17.015818 22374 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7919880ff1c45e08b99766137495c23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7919880ff1c45e08b99766137495c23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35353 } }
I20260927 12:45:17.016199 22374 leader_election.cc:302] T 00000000000000000000000000000000 P b7919880ff1c45e08b99766137495c23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b7919880ff1c45e08b99766137495c23; no voters:
I20260927 12:45:17.016521 22374 leader_election.cc:288] T 00000000000000000000000000000000 P b7919880ff1c45e08b99766137495c23 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:45:17.016810 22379 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7919880ff1c45e08b99766137495c23 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:45:17.017064 22379 raft_consensus.cc:686] T 00000000000000000000000000000000 P b7919880ff1c45e08b99766137495c23 [term 1 LEADER]: Becoming Leader. State: Replica: b7919880ff1c45e08b99766137495c23, State: Running, Role: LEADER
I20260927 12:45:17.017405 22379 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7919880ff1c45e08b99766137495c23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7919880ff1c45e08b99766137495c23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35353 } }
I20260927 12:45:17.017835 22374 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7919880ff1c45e08b99766137495c23 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:45:17.019395 22380 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7919880ff1c45e08b99766137495c23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b7919880ff1c45e08b99766137495c23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7919880ff1c45e08b99766137495c23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35353 } } }
I20260927 12:45:17.019515 22380 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7919880ff1c45e08b99766137495c23 [sys.catalog]: This master's current role is: LEADER
I20260927 12:45:17.019817 22381 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7919880ff1c45e08b99766137495c23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7919880ff1c45e08b99766137495c23. Latest consensus state: current_term: 1 leader_uuid: "b7919880ff1c45e08b99766137495c23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7919880ff1c45e08b99766137495c23" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35353 } } }
I20260927 12:45:17.019942 22381 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7919880ff1c45e08b99766137495c23 [sys.catalog]: This master's current role is: LEADER
I20260927 12:45:17.021548 22377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:17.006594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54185 (local address 127.0.98.62:35353)
0927 12:45:17.007041 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:17.007056 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:17.007092 (+ 36us) server_negotiation.cc:407] Connection header received
0927 12:45:17.007806 (+ 714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:17.007825 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:17.008158 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:17.008509 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:17.009321 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:17.010827 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:17.011669 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:17.012460 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:17.012634 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:45:17.015514 (+ 2880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:45:17.015519 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:45:17.015532 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:45:17.015559 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:45:17.017779 (+ 2220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:45:17.018326 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:45:17.018332 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:45:17.018338 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:45:17.018473 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:45:17.018962 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:45:17.018966 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:45:17.018970 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:45:17.019501 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:45:17.019703 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0927 12:45:17.020208 (+ 505us) server_negotiation.cc:299] Negotiation successful
0927 12:45:17.020718 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":133,"threads_started":1}
W20260927 12:45:17.023455 22394 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b7919880ff1c45e08b99766137495c23: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 12:45:17.023561 22394 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 12:45:17.023653 22388 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:45:17.024540 22388 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:45:17.024680 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 12:45:17.030630 22388 catalog_manager.cc:1347] Generated new cluster ID: 5ff68ec93a1f4eb8b0b868720ea22f82
I20260927 12:45:17.030695 22388 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:45:17.035765 22377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:17.026905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59011 (local address 127.0.98.62:35353)
0927 12:45:17.027012 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:17.027016 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:17.027277 (+ 261us) server_negotiation.cc:407] Connection header received
0927 12:45:17.027394 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:17.027397 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:17.027451 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:17.027535 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:17.028908 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:17.029448 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:17.030289 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:17.030554 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:17.030657 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:45:17.032380 (+ 1723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:45:17.032383 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:45:17.032386 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:45:17.032413 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:45:17.034118 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:45:17.034567 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:45:17.034571 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:45:17.034573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:45:17.034661 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:45:17.034924 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:45:17.034927 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:45:17.034929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:45:17.035137 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:45:17.035267 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 12:45:17.035479 (+ 212us) server_negotiation.cc:299] Negotiation successful
0927 12:45:17.035613 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 12:45:17.044049 22388 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:45:17.044989 22388 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:45:17.049877 22388 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b7919880ff1c45e08b99766137495c23: Generated new TSK 0
I20260927 12:45:17.050555 22388 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:45:17.055161 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 22306
2026-09-27T12:45:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1402 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790512840836255-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:45:17 dist-test-slave-bnc1 krb5kdc[22405](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:45:17 dist-test-slave-bnc1 krb5kdc[22405](info): set up 2 sockets
Sep 27 12:45:17 dist-test-slave-bnc1 krb5kdc[22405](info): commencing operation
krb5kdc: starting...
W20260927 12:45:18.113816 392 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790512840836255-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790512840836255-392-0/krb5kdc/test-user.keytab.
Sep 27 12:45:18 dist-test-slave-bnc1 krb5kdc[22405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513118, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:45:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:45:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:45:18.233160 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790512840836255-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790512840836255-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790512840836255-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40673
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40673
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:45:18.341512 22418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:45:18.341832 22418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:45:18.341915 22418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:45:18.345983 22418 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:45:18.346069 22418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:45:18.346107 22418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:45:18.346164 22418 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:45:18.346203 22418 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:45:18.351373 22418 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790512840836255-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790512840836255-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40673
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790512840836255-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:45:18.352394 22418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:45:18.353368 22418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:45:18.361758 22424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:45:18.361781 22426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:45:18.361958 22418 server_base.cc:1034] running on GCE node
W20260927 12:45:18.361761 22423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:45:18.362622 22418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:45:18.363811 22418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:45:18.365000 22418 hybrid_clock.cc:648] HybridClock initialized: now 1790513118364980 us; error 32 us; skew 500 ppm
Sep 27 12:45:18 dist-test-slave-bnc1 krb5kdc[22405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513118, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:45:18.367666 22418 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:45:18.368602 22418 webserver.cc:460] Webserver started at http://127.0.98.62:39711/ using document root <none> and password file <none>
I20260927 12:45:18.369165 22418 fs_manager.cc:359] Metadata directory not provided
I20260927 12:45:18.369257 22418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:45:18.369496 22418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:45:18.371572 22418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790512840836255-392-0/minicluster-data/master-0/data/instance:
uuid: "f007df93fbd24276bebb2c58563a8d76"
format_stamp: "Formatted at 2026-09-27 12:45:18 on dist-test-slave-bnc1"
server_key: "0c63e7b4369f9fa86793976b7d4853fe"
server_key_iv: "cf3765b91a81cafe010263e9b6f57e9b"
server_key_version: "encryptionkey@0"
I20260927 12:45:18.372141 22418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790512840836255-392-0/minicluster-data/master-0/wal/instance:
uuid: "f007df93fbd24276bebb2c58563a8d76"
format_stamp: "Formatted at 2026-09-27 12:45:18 on dist-test-slave-bnc1"
server_key: "0c63e7b4369f9fa86793976b7d4853fe"
server_key_iv: "cf3765b91a81cafe010263e9b6f57e9b"
server_key_version: "encryptionkey@0"
I20260927 12:45:18.375905 22418 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 12:45:18.378347 22433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:45:18.379601 22418 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 12:45:18.379748 22418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790512840836255-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790512840836255-392-0/minicluster-data/master-0/wal
uuid: "f007df93fbd24276bebb2c58563a8d76"
format_stamp: "Formatted at 2026-09-27 12:45:18 on dist-test-slave-bnc1"
server_key: "0c63e7b4369f9fa86793976b7d4853fe"
server_key_iv: "cf3765b91a81cafe010263e9b6f57e9b"
server_key_version: "encryptionkey@0"
I20260927 12:45:18.379868 22418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790512840836255-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790512840836255-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790512840836255-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:45:18.408111 22418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:45:18.411329 22418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:45:18.411543 22418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:45:18.418423 22418 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40673
I20260927 12:45:18.418447 22485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40673 every 8 connection(s)
I20260927 12:45:18.419481 22418 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:45:18.422297 22486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:45:18.427742 22486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f007df93fbd24276bebb2c58563a8d76: Bootstrap starting.
I20260927 12:45:18.429076 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 22418
I20260927 12:45:18.429229 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790512840836255-392-0/minicluster-data/master-0/wal/instance
I20260927 12:45:18.429535 392 external_mini_cluster.cc:1428] Setting key 2649cd9e1cb5b5824db9bd41576279d4
I20260927 12:45:18.430647 22486 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f007df93fbd24276bebb2c58563a8d76: Neither blocks nor log segments found. Creating new log.
I20260927 12:45:18.431540 22486 log.cc:824] T 00000000000000000000000000000000 P f007df93fbd24276bebb2c58563a8d76: Log is configured to *not* fsync() on all Append() calls
I20260927 12:45:18.433765 22486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f007df93fbd24276bebb2c58563a8d76: No bootstrap required, opened a new log
I20260927 12:45:18.438138 22486 raft_consensus.cc:348] T 00000000000000000000000000000000 P f007df93fbd24276bebb2c58563a8d76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f007df93fbd24276bebb2c58563a8d76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40673 } }
I20260927 12:45:18.438426 22486 raft_consensus.cc:374] T 00000000000000000000000000000000 P f007df93fbd24276bebb2c58563a8d76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:45:18.438526 22486 raft_consensus.cc:729] T 00000000000000000000000000000000 P f007df93fbd24276bebb2c58563a8d76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f007df93fbd24276bebb2c58563a8d76, State: Initialized, Role: FOLLOWER
I20260927 12:45:18.439008 22486 consensus_queue.cc:260] T 00000000000000000000000000000000 P f007df93fbd24276bebb2c58563a8d76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f007df93fbd24276bebb2c58563a8d76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40673 } }
I20260927 12:45:18.439157 22486 raft_consensus.cc:388] T 00000000000000000000000000000000 P f007df93fbd24276bebb2c58563a8d76 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:45:18.439219 22486 raft_consensus.cc:482] T 00000000000000000000000000000000 P f007df93fbd24276bebb2c58563a8d76 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:45:18.439301 22486 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f007df93fbd24276bebb2c58563a8d76 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:45:18.439600 22489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:18.430740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54141 (local address 127.0.98.62:40673)
0927 12:45:18.431253 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:18.431272 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:18.431317 (+ 45us) server_negotiation.cc:407] Connection header received
0927 12:45:18.432175 (+ 858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:18.432195 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:18.432638 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:18.433000 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:18.433406 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:18.434873 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:18.435748 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:18.436562 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:18.436749 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:45:18.437532 (+ 783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:45:18.437538 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:45:18.437555 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:45:18.438068 (+ 513us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:45:18.438139 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:45:18.438172 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:45:18.438272 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 12:45:18.438412 (+ 140us) server_negotiation.cc:299] Negotiation successful
0927 12:45:18.438638 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":177,"threads_started":1}
I20260927 12:45:18.440428 22486 raft_consensus.cc:504] T 00000000000000000000000000000000 P f007df93fbd24276bebb2c58563a8d76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f007df93fbd24276bebb2c58563a8d76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40673 } }
I20260927 12:45:18.440804 22486 leader_election.cc:302] T 00000000000000000000000000000000 P f007df93fbd24276bebb2c58563a8d76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f007df93fbd24276bebb2c58563a8d76; no voters:
I20260927 12:45:18.441123 22486 leader_election.cc:288] T 00000000000000000000000000000000 P f007df93fbd24276bebb2c58563a8d76 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:45:18.441224 22491 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f007df93fbd24276bebb2c58563a8d76 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:45:18.441445 22491 raft_consensus.cc:686] T 00000000000000000000000000000000 P f007df93fbd24276bebb2c58563a8d76 [term 1 LEADER]: Becoming Leader. State: Replica: f007df93fbd24276bebb2c58563a8d76, State: Running, Role: LEADER
I20260927 12:45:18.441846 22491 consensus_queue.cc:237] T 00000000000000000000000000000000 P f007df93fbd24276bebb2c58563a8d76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f007df93fbd24276bebb2c58563a8d76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40673 } }
I20260927 12:45:18.442528 22486 sys_catalog.cc:564] T 00000000000000000000000000000000 P f007df93fbd24276bebb2c58563a8d76 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:45:18.442731 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 12:45:18.443694 22492 sys_catalog.cc:455] T 00000000000000000000000000000000 P f007df93fbd24276bebb2c58563a8d76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f007df93fbd24276bebb2c58563a8d76. Latest consensus state: current_term: 1 leader_uuid: "f007df93fbd24276bebb2c58563a8d76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f007df93fbd24276bebb2c58563a8d76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40673 } } }
I20260927 12:45:18.443817 22492 sys_catalog.cc:458] T 00000000000000000000000000000000 P f007df93fbd24276bebb2c58563a8d76 [sys.catalog]: This master's current role is: LEADER
I20260927 12:45:18.444211 22493 sys_catalog.cc:455] T 00000000000000000000000000000000 P f007df93fbd24276bebb2c58563a8d76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f007df93fbd24276bebb2c58563a8d76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f007df93fbd24276bebb2c58563a8d76" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40673 } } }
I20260927 12:45:18.444351 22493 sys_catalog.cc:458] T 00000000000000000000000000000000 P f007df93fbd24276bebb2c58563a8d76 [sys.catalog]: This master's current role is: LEADER
W20260927 12:45:18.447790 22510 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f007df93fbd24276bebb2c58563a8d76: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 12:45:18.447870 22510 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 12:45:18.447926 22500 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:45:18.448776 22500 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:45:18.450176 22489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:18.445046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53655 (local address 127.0.98.62:40673)
0927 12:45:18.445157 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:18.445161 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:18.445718 (+ 557us) server_negotiation.cc:407] Connection header received
0927 12:45:18.445894 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:18.445897 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:18.445957 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:18.446009 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:18.446414 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:18.447155 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:18.448254 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:18.448579 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:18.448655 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:45:18.449678 (+ 1023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:45:18.449684 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:45:18.449687 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:45:18.449699 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:45:18.449729 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:45:18.449743 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:45:18.449828 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 12:45:18.449954 (+ 126us) server_negotiation.cc:299] Negotiation successful
0927 12:45:18.450017 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 12:45:18.455183 22500 catalog_manager.cc:1347] Generated new cluster ID: ce10f27d1a94482c88524211855e1c32
I20260927 12:45:18.455253 22500 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:45:18.472982 22500 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:45:18.474103 22500 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:45:18.490566 22500 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f007df93fbd24276bebb2c58563a8d76: Generated new TSK 0
I20260927 12:45:18.491372 22500 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:45:18.503362 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 22418
2026-09-27T12:45:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1448 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790512840836255-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:45:18 dist-test-slave-bnc1 krb5kdc[22517](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:45:18 dist-test-slave-bnc1 krb5kdc[22517](info): set up 2 sockets
Sep 27 12:45:18 dist-test-slave-bnc1 krb5kdc[22517](info): commencing operation
krb5kdc: starting...
W20260927 12:45:19.553804 392 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790512840836255-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790512840836255-392-0/krb5kdc/test-user.keytab.
Sep 27 12:45:19 dist-test-slave-bnc1 krb5kdc[22517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:45:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:45:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:45:19.672760 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790512840836255-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790512840836255-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790512840836255-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42239
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42239
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:45:19.780161 22530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:45:19.780488 22530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:45:19.780602 22530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:45:19.784552 22530 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:45:19.784664 22530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:45:19.784709 22530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:45:19.784763 22530 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:45:19.784801 22530 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:45:19.790032 22530 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790512840836255-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790512840836255-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790512840836255-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:45:19.791115 22530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:45:19.792142 22530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 12:45:19.800827 22530 server_base.cc:1034] running on GCE node
W20260927 12:45:19.800786 22538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:45:19.800786 22535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:45:19.800789 22536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:45:19.801746 22530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:45:19.802860 22530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:45:19.804064 22530 hybrid_clock.cc:648] HybridClock initialized: now 1790513119804039 us; error 38 us; skew 500 ppm
Sep 27 12:45:19 dist-test-slave-bnc1 krb5kdc[22517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513119, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:45:19.813539 22530 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:45:19.814626 22530 webserver.cc:460] Webserver started at http://127.0.98.62:43461/ using document root <none> and password file <none>
I20260927 12:45:19.815169 22530 fs_manager.cc:359] Metadata directory not provided
I20260927 12:45:19.815229 22530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:45:19.815407 22530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:45:19.817548 22530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790512840836255-392-0/minicluster-data/master-0/data/instance:
uuid: "d9c7b73b86dc432083630a4179f40939"
format_stamp: "Formatted at 2026-09-27 12:45:19 on dist-test-slave-bnc1"
server_key: "02f542da86ac50b1e4f5ab06d79ebff4"
server_key_iv: "4924d015c25b8c6b7db768da9c2a502b"
server_key_version: "encryptionkey@0"
I20260927 12:45:19.818243 22530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790512840836255-392-0/minicluster-data/master-0/wal/instance:
uuid: "d9c7b73b86dc432083630a4179f40939"
format_stamp: "Formatted at 2026-09-27 12:45:19 on dist-test-slave-bnc1"
server_key: "02f542da86ac50b1e4f5ab06d79ebff4"
server_key_iv: "4924d015c25b8c6b7db768da9c2a502b"
server_key_version: "encryptionkey@0"
I20260927 12:45:19.822222 22530 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 12:45:19.824930 22545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:45:19.826570 22530 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:45:19.826714 22530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790512840836255-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790512840836255-392-0/minicluster-data/master-0/wal
uuid: "d9c7b73b86dc432083630a4179f40939"
format_stamp: "Formatted at 2026-09-27 12:45:19 on dist-test-slave-bnc1"
server_key: "02f542da86ac50b1e4f5ab06d79ebff4"
server_key_iv: "4924d015c25b8c6b7db768da9c2a502b"
server_key_version: "encryptionkey@0"
I20260927 12:45:19.826840 22530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790512840836255-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790512840836255-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790512840836255-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:45:19.831398 22530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:45:19.834611 22530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:45:19.834753 22530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:45:19.841434 22530 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42239
I20260927 12:45:19.841503 22597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42239 every 8 connection(s)
I20260927 12:45:19.842425 22530 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:45:19.845165 22598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:45:19.848631 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 22530
I20260927 12:45:19.848742 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790512840836255-392-0/minicluster-data/master-0/wal/instance
I20260927 12:45:19.849030 392 external_mini_cluster.cc:1428] Setting key 28df68f0ac867a9bcedf812cfdb495de
I20260927 12:45:19.851293 22598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9c7b73b86dc432083630a4179f40939: Bootstrap starting.
I20260927 12:45:19.854017 22598 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d9c7b73b86dc432083630a4179f40939: Neither blocks nor log segments found. Creating new log.
I20260927 12:45:19.854570 22601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:19.850437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45335 (local address 127.0.98.62:42239)
0927 12:45:19.850863 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:19.850882 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:19.850925 (+ 43us) server_negotiation.cc:407] Connection header received
0927 12:45:19.851726 (+ 801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:19.851749 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:19.852136 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:19.852509 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:19.853149 (+ 640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:45:19.853154 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:45:19.853168 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:45:19.853177 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:45:19.853233 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:45:19.853258 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:45:19.853357 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 12:45:19.853475 (+ 118us) server_negotiation.cc:299] Negotiation successful
0927 12:45:19.853654 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":104,"threads_started":1}
I20260927 12:45:19.854878 22598 log.cc:824] T 00000000000000000000000000000000 P d9c7b73b86dc432083630a4179f40939: Log is configured to *not* fsync() on all Append() calls
I20260927 12:45:19.856815 22598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9c7b73b86dc432083630a4179f40939: No bootstrap required, opened a new log
I20260927 12:45:19.857236 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 12:45:19.860905 22601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:19.859287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58013 (local address 127.0.98.62:42239)
0927 12:45:19.859418 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:19.859422 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:19.859653 (+ 231us) server_negotiation.cc:407] Connection header received
0927 12:45:19.859743 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:19.859747 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:19.859789 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:19.859895 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:19.860571 (+ 676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:45:19.860575 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:45:19.860578 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 12:45:19.860589 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:45:19.860616 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 12:45:19.860628 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:45:19.860724 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 12:45:19.860763 (+ 39us) server_negotiation.cc:299] Negotiation successful
0927 12:45:19.860798 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 12:45:19.860725 22598 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9c7b73b86dc432083630a4179f40939 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9c7b73b86dc432083630a4179f40939" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42239 } }
I20260927 12:45:19.860978 22598 raft_consensus.cc:374] T 00000000000000000000000000000000 P d9c7b73b86dc432083630a4179f40939 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 12:45:19.861028 22598 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9c7b73b86dc432083630a4179f40939 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9c7b73b86dc432083630a4179f40939, State: Initialized, Role: FOLLOWER
I20260927 12:45:19.861454 22598 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9c7b73b86dc432083630a4179f40939 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9c7b73b86dc432083630a4179f40939" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42239 } }
I20260927 12:45:19.861622 22598 raft_consensus.cc:388] T 00000000000000000000000000000000 P d9c7b73b86dc432083630a4179f40939 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 12:45:19.861716 22598 raft_consensus.cc:482] T 00000000000000000000000000000000 P d9c7b73b86dc432083630a4179f40939 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 12:45:19.861831 22598 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9c7b73b86dc432083630a4179f40939 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:45:19.862697 22598 raft_consensus.cc:504] T 00000000000000000000000000000000 P d9c7b73b86dc432083630a4179f40939 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9c7b73b86dc432083630a4179f40939" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42239 } }
I20260927 12:45:19.863025 22598 leader_election.cc:302] T 00000000000000000000000000000000 P d9c7b73b86dc432083630a4179f40939 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d9c7b73b86dc432083630a4179f40939; no voters:
I20260927 12:45:19.863309 22598 leader_election.cc:288] T 00000000000000000000000000000000 P d9c7b73b86dc432083630a4179f40939 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 12:45:19.863426 22608 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d9c7b73b86dc432083630a4179f40939 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:45:19.863667 22608 raft_consensus.cc:686] T 00000000000000000000000000000000 P d9c7b73b86dc432083630a4179f40939 [term 1 LEADER]: Becoming Leader. State: Replica: d9c7b73b86dc432083630a4179f40939, State: Running, Role: LEADER
I20260927 12:45:19.864001 22608 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9c7b73b86dc432083630a4179f40939 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9c7b73b86dc432083630a4179f40939" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42239 } }
I20260927 12:45:19.864557 22598 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9c7b73b86dc432083630a4179f40939 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:45:19.865732 22610 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9c7b73b86dc432083630a4179f40939 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9c7b73b86dc432083630a4179f40939. Latest consensus state: current_term: 1 leader_uuid: "d9c7b73b86dc432083630a4179f40939" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9c7b73b86dc432083630a4179f40939" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42239 } } }
I20260927 12:45:19.865875 22610 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9c7b73b86dc432083630a4179f40939 [sys.catalog]: This master's current role is: LEADER
I20260927 12:45:19.866120 22609 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9c7b73b86dc432083630a4179f40939 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d9c7b73b86dc432083630a4179f40939" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9c7b73b86dc432083630a4179f40939" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42239 } } }
I20260927 12:45:19.866209 22609 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9c7b73b86dc432083630a4179f40939 [sys.catalog]: This master's current role is: LEADER
I20260927 12:45:19.866258 22617 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:45:19.869060 22617 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:45:19.874663 22617 catalog_manager.cc:1347] Generated new cluster ID: dd099cc1fbf448349827e10aed86c791
I20260927 12:45:19.874737 22617 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:45:19.881237 22617 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:45:19.881587 22617 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:45:19.887995 22617 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d9c7b73b86dc432083630a4179f40939: Generated new TSK 0
I20260927 12:45:19.888818 22617 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 12:45:19.900838 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 22530
2026-09-27T12:45:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1397 ms)
[----------] 18 tests from AuthTokenIssuingTest (25485 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512840836255-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:45:19 dist-test-slave-bnc1 krb5kdc[22629](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:45:19 dist-test-slave-bnc1 krb5kdc[22629](info): set up 2 sockets
Sep 27 12:45:19 dist-test-slave-bnc1 krb5kdc[22629](info): commencing operation
krb5kdc: starting...
W20260927 12:45:20.957955 392 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512840836255-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512840836255-392-0/krb5kdc/test-user.keytab.
Sep 27 12:45:21 dist-test-slave-bnc1 krb5kdc[22629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513121, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:45:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:45:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:45:21.080818 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512840836255-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512840836255-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512840836255-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42777
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42777,127.0.98.61:43239,127.0.98.60:35603
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:45:21.191511 22642 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 12:45:21.191797 22642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:45:21.191819 22642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:45:21.191884 22642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:45:21.195869 22642 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:45:21.195920 22642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:45:21.195940 22642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:45:21.195955 22642 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:45:21.196012 22642 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:45:21.201192 22642 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512840836255-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512840836255-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42777,127.0.98.61:43239,127.0.98.60:35603
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22642
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512840836255-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:45:21.202215 22642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:45:21.203230 22642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:45:21.211639 22650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:45:21.211653 22647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:45:21.211660 22648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:45:21.211772 22642 server_base.cc:1034] running on GCE node
I20260927 12:45:21.212551 22642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:45:21.213671 22642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:45:21.214843 22642 hybrid_clock.cc:648] HybridClock initialized: now 1790513121214822 us; error 31 us; skew 500 ppm
Sep 27 12:45:21 dist-test-slave-bnc1 krb5kdc[22629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513121, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:45:21.217775 22642 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:45:21.218667 22642 webserver.cc:460] Webserver started at http://127.0.98.62:40301/ using document root <none> and password file <none>
I20260927 12:45:21.219213 22642 fs_manager.cc:359] Metadata directory not provided
I20260927 12:45:21.219272 22642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:45:21.219520 22642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:45:21.221542 22642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512840836255-392-0/minicluster-data/master-0/data/instance:
uuid: "109924a8231d4bfc9c6042b994a5434c"
format_stamp: "Formatted at 2026-09-27 12:45:21 on dist-test-slave-bnc1"
server_key: "602b800836e823f6111b2f9653793938"
server_key_iv: "8886d023b39b76f4b4618aa0020d8648"
server_key_version: "encryptionkey@0"
I20260927 12:45:21.222167 22642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512840836255-392-0/minicluster-data/master-0/wal/instance:
uuid: "109924a8231d4bfc9c6042b994a5434c"
format_stamp: "Formatted at 2026-09-27 12:45:21 on dist-test-slave-bnc1"
server_key: "602b800836e823f6111b2f9653793938"
server_key_iv: "8886d023b39b76f4b4618aa0020d8648"
server_key_version: "encryptionkey@0"
I20260927 12:45:21.225762 22642 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:45:21.228092 22657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:45:21.229322 22642 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:45:21.229449 22642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512840836255-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512840836255-392-0/minicluster-data/master-0/wal
uuid: "109924a8231d4bfc9c6042b994a5434c"
format_stamp: "Formatted at 2026-09-27 12:45:21 on dist-test-slave-bnc1"
server_key: "602b800836e823f6111b2f9653793938"
server_key_iv: "8886d023b39b76f4b4618aa0020d8648"
server_key_version: "encryptionkey@0"
I20260927 12:45:21.229549 22642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512840836255-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512840836255-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512840836255-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:45:21.244875 22642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:45:21.247753 22642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:45:21.247960 22642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:45:21.255090 22642 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42777
I20260927 12:45:21.255379 22709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42777 every 8 connection(s)
I20260927 12:45:21.256124 22642 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:45:21.256588 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 22642
I20260927 12:45:21.256752 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512840836255-392-0/minicluster-data/master-0/wal/instance
I20260927 12:45:21.257030 392 external_mini_cluster.cc:1428] Setting key 4a01aa221cc209dc3b3105bc79531312
I20260927 12:45:21.259037 22710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:45:21.264544 22710 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42777 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 27 12:45:21 dist-test-slave-bnc1 krb5kdc[22629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513121, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:45:21.281888 22714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:21.267335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58718 (local address 127.0.98.62:42777)
0927 12:45:21.267891 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:21.267911 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:21.268091 (+ 180us) server_negotiation.cc:407] Connection header received
0927 12:45:21.269369 (+ 1278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:21.269374 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:21.269551 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:21.269769 (+ 218us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:21.270727 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:21.271881 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:21.273459 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:21.273898 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:21.274030 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:45:21.276680 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:45:21.276686 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:45:21.276697 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:45:21.276717 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:45:21.278501 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:45:21.279027 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:45:21.279033 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:45:21.279035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:45:21.279141 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:45:21.279455 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:45:21.279458 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:45:21.279459 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:45:21.279889 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:45:21.280022 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 12:45:21.280466 (+ 444us) server_negotiation.cc:299] Negotiation successful
0927 12:45:21.280728 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"spinlock_wait_cycles":76032,"thread_start_us":209,"threads_started":1}
I20260927 12:45:21.281908 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:21.267335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42777 (local address 127.0.0.1:58718)
0927 12:45:21.267943 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0927 12:45:21.267988 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 12:45:21.269046 (+ 1058us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:45:21.269817 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:45:21.269826 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:45:21.270320 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:45:21.270573 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:45:21.270586 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:45:21.272013 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:45:21.272017 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:45:21.273335 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:45:21.273348 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:45:21.274079 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:45:21.274085 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:45:21.274246 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:45:21.274872 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:45:21.274899 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:45:21.276529 (+ 1630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:45:21.278618 (+ 2089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:45:21.278627 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:45:21.278637 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:45:21.278877 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:45:21.279237 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:45:21.279240 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:45:21.279242 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:45:21.279355 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:45:21.280004 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:45:21.280010 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:45:21.280153 (+ 143us) client_negotiation.cc:769] Sending connection context
0927 12:45:21.280401 (+ 248us) client_negotiation.cc:240] Negotiation successful
0927 12:45:21.280813 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"mutex_wait_us":113,"thread_start_us":147,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512840836255-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512840836255-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260927 12:45:21.284493 22710 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43239 } has no permanent_uuid. Determining permanent_uuid...
I20260927 12:45:21.285241 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:21.284975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43239 (local address 127.0.0.1:51848)
0927 12:45:21.285099 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 12:45:21.285178 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 12:45:21.285432 22659 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43239: connect: Connection refused (error 111)
I20260927 12:45:21.285465 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512840836255-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512840836255-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512840836255-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512840836255-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43239
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512840836255-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42777,127.0.98.61:43239,127.0.98.60:35603
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 12:45:21.286330 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:21.286120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43239 (local address 127.0.0.1:51852)
0927 12:45:21.286229 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 12:45:21.286275 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 12:45:21.286458 22710 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43239: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43239: connect: Connection refused (error 111)
I20260927 12:45:21.335700 22710 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43239 } attempt: 1
I20260927 12:45:21.336594 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:21.336288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43239 (local address 127.0.0.1:51856)
0927 12:45:21.336450 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 12:45:21.336516 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260927 12:45:21.337327 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:21.337120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43239 (local address 127.0.0.1:51864)
0927 12:45:21.337224 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0927 12:45:21.337275 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 12:45:21.337451 22710 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43239: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43239: connect: Connection refused (error 111)
W20260927 12:45:21.396878 22717 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 12:45:21.397239 22717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:45:21.397310 22717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:45:21.397403 22717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:45:21.401484 22717 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:45:21.401602 22717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:45:21.401650 22717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:45:21.401686 22717 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:45:21.401719 22717 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:45:21.405699 22710 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43239 } attempt: 2
I20260927 12:45:21.406535 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:21.406231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43239 (local address 127.0.0.1:55770)
0927 12:45:21.406389 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 12:45:21.406462 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260927 12:45:21.407239 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:21.407019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43239 (local address 127.0.0.1:55778)
0927 12:45:21.407128 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0927 12:45:21.407175 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260927 12:45:21.407207 22717 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512840836255-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512840836255-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42777,127.0.98.61:43239,127.0.98.60:35603
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512840836255-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512840836255-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22717
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512840836255-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
W20260927 12:45:21.407371 22710 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43239: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43239: connect: Connection refused (error 111)
I20260927 12:45:21.408342 22717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:45:21.409425 22717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:45:21.417928 22725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:45:21.417990 22723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:45:21.418179 22717 server_base.cc:1034] running on GCE node
W20260927 12:45:21.418179 22722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:45:21.418830 22717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:45:21.419970 22717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:45:21.421137 22717 hybrid_clock.cc:648] HybridClock initialized: now 1790513121421091 us; error 59 us; skew 500 ppm
Sep 27 12:45:21 dist-test-slave-bnc1 krb5kdc[22629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513121, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:45:21.423933 22717 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260927 12:45:21.424873 22717 webserver.cc:460] Webserver started at http://127.0.98.61:43033/ using document root <none> and password file <none>
I20260927 12:45:21.425431 22717 fs_manager.cc:359] Metadata directory not provided
I20260927 12:45:21.425491 22717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:45:21.425760 22717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:45:21.427835 22717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512840836255-392-0/minicluster-data/master-1/data/instance:
uuid: "73b52216271a44559be540b85a7a2f58"
format_stamp: "Formatted at 2026-09-27 12:45:21 on dist-test-slave-bnc1"
server_key: "66cee9c9e8d3d5e8c9059776bfda7363"
server_key_iv: "ba6c25e6f38938bade7aeab1f98e5cbc"
server_key_version: "encryptionkey@0"
I20260927 12:45:21.428403 22717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512840836255-392-0/minicluster-data/master-1/wal/instance:
uuid: "73b52216271a44559be540b85a7a2f58"
format_stamp: "Formatted at 2026-09-27 12:45:21 on dist-test-slave-bnc1"
server_key: "66cee9c9e8d3d5e8c9059776bfda7363"
server_key_iv: "ba6c25e6f38938bade7aeab1f98e5cbc"
server_key_version: "encryptionkey@0"
I20260927 12:45:21.432205 22717 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 12:45:21.434718 22732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:45:21.435914 22717 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 12:45:21.436061 22717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512840836255-392-0/minicluster-data/master-1/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512840836255-392-0/minicluster-data/master-1/wal
uuid: "73b52216271a44559be540b85a7a2f58"
format_stamp: "Formatted at 2026-09-27 12:45:21 on dist-test-slave-bnc1"
server_key: "66cee9c9e8d3d5e8c9059776bfda7363"
server_key_iv: "ba6c25e6f38938bade7aeab1f98e5cbc"
server_key_version: "encryptionkey@0"
I20260927 12:45:21.436173 22717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512840836255-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512840836255-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512840836255-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:45:21.461977 22717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:45:21.464951 22717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:45:21.465163 22717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:45:21.472157 22717 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:43239
I20260927 12:45:21.472438 22784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:43239 every 8 connection(s)
I20260927 12:45:21.473165 22717 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512840836255-392-0/minicluster-data/master-1/data/info.pb
I20260927 12:45:21.475893 22785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:45:21.479677 22785 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42777 } has no permanent_uuid. Determining permanent_uuid...
I20260927 12:45:21.481947 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 22717
I20260927 12:45:21.482080 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512840836255-392-0/minicluster-data/master-1/wal/instance
I20260927 12:45:21.482419 392 external_mini_cluster.cc:1428] Setting key 4ce4c3e3c2f9ffc2e32fbd5c95f05949
Sep 27 12:45:21 dist-test-slave-bnc1 krb5kdc[22629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513121, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260927 12:45:21.493865 22714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:21.481607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53786 (local address 127.0.98.62:42777)
0927 12:45:21.481785 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:21.481789 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:21.482872 (+ 1083us) server_negotiation.cc:407] Connection header received
0927 12:45:21.483843 (+ 971us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:21.483847 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:21.483903 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:21.484012 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:21.485179 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:21.485806 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:21.486964 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:21.487198 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:21.487333 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:45:21.490085 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:45:21.490088 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:45:21.490090 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:45:21.490116 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:45:21.491491 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:45:21.492141 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:45:21.492147 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:45:21.492151 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:45:21.492236 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:45:21.492644 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:45:21.492648 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:45:21.492649 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:45:21.492850 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:45:21.492964 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 12:45:21.493531 (+ 567us) server_negotiation.cc:299] Negotiation successful
0927 12:45:21.493690 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 12:45:21.494839 22787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:21.482114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42777 (local address 127.0.0.1:53786)
0927 12:45:21.482699 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0927 12:45:21.482744 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 12:45:21.483609 (+ 865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:45:21.484187 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:45:21.484195 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:45:21.484544 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:45:21.485028 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:45:21.485041 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:45:21.485951 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:45:21.485954 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:45:21.486822 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:45:21.486831 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:45:21.487388 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:45:21.487392 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:45:21.487559 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:45:21.488194 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:45:21.488209 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:45:21.489881 (+ 1672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:45:21.491617 (+ 1736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:45:21.491623 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:45:21.491640 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:45:21.492021 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:45:21.492369 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:45:21.492372 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:45:21.492374 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:45:21.492554 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:45:21.492963 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:45:21.492970 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:45:21.493261 (+ 291us) client_negotiation.cc:769] Sending connection context
0927 12:45:21.493465 (+ 204us) client_negotiation.cc:240] Negotiation successful
0927 12:45:21.493868 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":181,"threads_started":1}
I20260927 12:45:21.496368 22785 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43239 } has no permanent_uuid. Determining permanent_uuid...
I20260927 12:45:21.498519 22710 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43239 } attempt: 3
Sep 27 12:45:21 dist-test-slave-bnc1 krb5kdc[22629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513121, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 27 12:45:21 dist-test-slave-bnc1 krb5kdc[22629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513121, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512840836255-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512840836255-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260927 12:45:21.508947 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:21.498936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43239 (local address 127.0.0.1:55784)
0927 12:45:21.499250 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0927 12:45:21.499265 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:45:21.499350 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:45:21.499580 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:45:21.499583 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:45:21.499746 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:45:21.499893 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:45:21.499902 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:45:21.501051 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:45:21.501057 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:45:21.501815 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:45:21.501825 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:45:21.502513 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:45:21.502519 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:45:21.502636 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:45:21.503143 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:45:21.503164 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:45:21.504590 (+ 1426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:45:21.507227 (+ 2637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:45:21.507230 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:45:21.507231 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:45:21.507475 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:45:21.507789 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:45:21.507792 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:45:21.507794 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:45:21.507877 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:45:21.508440 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:45:21.508446 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:45:21.508542 (+ 96us) client_negotiation.cc:769] Sending connection context
0927 12:45:21.508647 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 12:45:21.508781 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234}
I20260927 12:45:21.508947 22787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:21.496962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43239 (local address 127.0.0.1:55780)
0927 12:45:21.497062 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0927 12:45:21.497075 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:45:21.497153 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:45:21.497990 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:45:21.497993 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:45:21.498135 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:45:21.498224 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:45:21.498229 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:45:21.499861 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:45:21.499865 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:45:21.500464 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:45:21.500472 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:45:21.501088 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:45:21.501092 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:45:21.501202 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:45:21.501741 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:45:21.501753 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:45:21.504060 (+ 2307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:45:21.507291 (+ 3231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:45:21.507296 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:45:21.507301 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:45:21.507546 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:45:21.507862 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:45:21.507868 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:45:21.507873 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:45:21.507963 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:45:21.508459 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:45:21.508462 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:45:21.508536 (+ 74us) client_negotiation.cc:769] Sending connection context
0927 12:45:21.508620 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 12:45:21.508762 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260927 12:45:21.509081 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512840836255-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512840836255-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512840836255-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512840836255-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35603
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512840836255-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42777,127.0.98.61:43239,127.0.98.60:35603
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 12:45:21.509150 22791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:21.498985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55784 (local address 127.0.98.61:43239)
0927 12:45:21.499307 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:21.499316 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:21.499329 (+ 13us) server_negotiation.cc:407] Connection header received
0927 12:45:21.499426 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:21.499432 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:21.499474 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:21.499565 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:21.500016 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:21.500916 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:21.501984 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:21.502383 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:21.502466 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:45:21.504756 (+ 2290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:45:21.504762 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:45:21.504765 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:45:21.504792 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:45:21.507115 (+ 2323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:45:21.507593 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:45:21.507600 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:45:21.507604 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:45:21.507685 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:45:21.507970 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:45:21.507975 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:45:21.507976 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:45:21.508321 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:45:21.508478 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 12:45:21.508755 (+ 277us) server_negotiation.cc:299] Negotiation successful
0927 12:45:21.508936 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":239,"thread_start_us":125,"threads_started":1}
I20260927 12:45:21.509614 22790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:21.497108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55780 (local address 127.0.98.61:43239)
0927 12:45:21.497473 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:21.497488 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:21.497516 (+ 28us) server_negotiation.cc:407] Connection header received
0927 12:45:21.497593 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:21.497603 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:21.497850 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:21.498013 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:21.498350 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:21.499731 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:21.500578 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:21.500962 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:21.501089 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:45:21.504192 (+ 3103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:45:21.504200 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:45:21.504213 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:45:21.504233 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:45:21.507182 (+ 2949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:45:21.507645 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:45:21.507651 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:45:21.507653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:45:21.507739 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:45:21.508064 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:45:21.508069 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:45:21.508074 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:45:21.508359 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:45:21.509163 (+ 804us) server_negotiation.cc:1035] Waiting for connection context
0927 12:45:21.509252 (+ 89us) server_negotiation.cc:299] Negotiation successful
0927 12:45:21.509392 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":240,"thread_start_us":128,"threads_started":1}
I20260927 12:45:21.511325 22785 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35603 } has no permanent_uuid. Determining permanent_uuid...
I20260927 12:45:21.511368 22710 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35603 } has no permanent_uuid. Determining permanent_uuid...
I20260927 12:45:21.512032 22787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:21.511778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35603 (local address 127.0.0.1:34568)
0927 12:45:21.511895 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0927 12:45:21.511970 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 12:45:21.512209 22734 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35603: connect: Connection refused (error 111)
I20260927 12:45:21.512082 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:21.511800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35603 (local address 127.0.0.1:34564)
0927 12:45:21.511968 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 12:45:21.512012 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260927 12:45:21.513011 22787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:21.512803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35603 (local address 127.0.0.1:34582)
0927 12:45:21.512906 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0927 12:45:21.512954 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 12:45:21.513300 22785 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35603: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35603: connect: Connection refused (error 111)
I20260927 12:45:21.513710 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:21.513338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35603 (local address 127.0.0.1:34594)
0927 12:45:21.513595 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0927 12:45:21.513642 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":85,"threads_started":1}
W20260927 12:45:21.513832 22710 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35603: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35603: connect: Connection refused (error 111)
I20260927 12:45:21.544993 22710 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35603 } attempt: 1
I20260927 12:45:21.545843 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:21.545530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35603 (local address 127.0.0.1:34596)
0927 12:45:21.545704 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 12:45:21.545770 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260927 12:45:21.546540 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:21.546333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35603 (local address 127.0.0.1:34610)
0927 12:45:21.546441 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0927 12:45:21.546489 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 12:45:21.546669 22710 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35603: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35603: connect: Connection refused (error 111)
I20260927 12:45:21.562549 22785 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35603 } attempt: 1
I20260927 12:45:21.563434 22787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:21.563075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35603 (local address 127.0.0.1:34612)
0927 12:45:21.563252 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 12:45:21.563338 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260927 12:45:21.564152 22787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:21.563933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35603 (local address 127.0.0.1:34624)
0927 12:45:21.564038 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0927 12:45:21.564089 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260927 12:45:21.564388 22785 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35603: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35603: connect: Connection refused (error 111)
I20260927 12:45:21.621829 22710 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35603 } attempt: 2
I20260927 12:45:21.622645 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:21.622355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35603 (local address 127.0.0.1:34638)
0927 12:45:21.622497 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 12:45:21.622569 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260927 12:45:21.622542 22792 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 12:45:21.622937 22792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:45:21.623003 22792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:45:21.623077 22792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260927 12:45:21.623394 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:21.623165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35603 (local address 127.0.0.1:34654)
0927 12:45:21.623285 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0927 12:45:21.623335 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 12:45:21.623518 22710 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35603: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35603: connect: Connection refused (error 111)
W20260927 12:45:21.627391 22792 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:45:21.627475 22792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:45:21.627516 22792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:45:21.627576 22792 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:45:21.627616 22792 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:45:21.632611 22785 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35603 } attempt: 2
I20260927 12:45:21.633160 22792 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512840836255-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512840836255-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42777,127.0.98.61:43239,127.0.98.60:35603
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512840836255-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35603
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512840836255-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22792
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512840836255-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:45:21.633435 22787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:21.633114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35603 (local address 127.0.0.1:34670)
0927 12:45:21.633273 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 12:45:21.633349 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 12:45:21.634203 22787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:21.634002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35603 (local address 127.0.0.1:34672)
0927 12:45:21.634090 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0927 12:45:21.634142 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260927 12:45:21.634339 22785 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35603: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35603: connect: Connection refused (error 111)
I20260927 12:45:21.634577 22792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:45:21.635532 22792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:45:21.644047 22802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:45:21.644006 22799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:45:21.644285 22792 server_base.cc:1034] running on GCE node
W20260927 12:45:21.644017 22800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:45:21.644870 22792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:45:21.646013 22792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:45:21.647189 22792 hybrid_clock.cc:648] HybridClock initialized: now 1790513121647171 us; error 27 us; skew 500 ppm
Sep 27 12:45:21 dist-test-slave-bnc1 krb5kdc[22629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513121, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:45:21.649999 22792 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260927 12:45:21.650840 22792 webserver.cc:460] Webserver started at http://127.0.98.60:39137/ using document root <none> and password file <none>
I20260927 12:45:21.651396 22792 fs_manager.cc:359] Metadata directory not provided
I20260927 12:45:21.651487 22792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:45:21.651712 22792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:45:21.653805 22792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512840836255-392-0/minicluster-data/master-2/data/instance:
uuid: "7b4f6ce0f00747018d4095ba9b32a246"
format_stamp: "Formatted at 2026-09-27 12:45:21 on dist-test-slave-bnc1"
server_key: "5908531ff354ac819a0a04bd260618f6"
server_key_iv: "f557e7eaa5a496f697bba6ef919c7cd1"
server_key_version: "encryptionkey@0"
I20260927 12:45:21.654403 22792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512840836255-392-0/minicluster-data/master-2/wal/instance:
uuid: "7b4f6ce0f00747018d4095ba9b32a246"
format_stamp: "Formatted at 2026-09-27 12:45:21 on dist-test-slave-bnc1"
server_key: "5908531ff354ac819a0a04bd260618f6"
server_key_iv: "f557e7eaa5a496f697bba6ef919c7cd1"
server_key_version: "encryptionkey@0"
I20260927 12:45:21.658049 22792 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:45:21.660419 22809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:45:21.661654 22792 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 12:45:21.661792 22792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512840836255-392-0/minicluster-data/master-2/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512840836255-392-0/minicluster-data/master-2/wal
uuid: "7b4f6ce0f00747018d4095ba9b32a246"
format_stamp: "Formatted at 2026-09-27 12:45:21 on dist-test-slave-bnc1"
server_key: "5908531ff354ac819a0a04bd260618f6"
server_key_iv: "f557e7eaa5a496f697bba6ef919c7cd1"
server_key_version: "encryptionkey@0"
I20260927 12:45:21.661909 22792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512840836255-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512840836255-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512840836255-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:45:21.680424 22792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:45:21.683300 22792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:45:21.683513 22792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:45:21.690357 22792 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:35603
I20260927 12:45:21.690667 22861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:35603 every 8 connection(s)
I20260927 12:45:21.691335 22792 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512840836255-392-0/minicluster-data/master-2/data/info.pb
I20260927 12:45:21.694039 22862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:45:21.696352 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 22792
I20260927 12:45:21.696496 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790512840836255-392-0/minicluster-data/master-2/wal/instance
I20260927 12:45:21.696770 392 external_mini_cluster.cc:1428] Setting key 73227935d97e86abb0202e970c2c32dc
I20260927 12:45:21.698076 22862 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42777 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 12:45:21 dist-test-slave-bnc1 krb5kdc[22629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513121, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 27 12:45:21 dist-test-slave-bnc1 krb5kdc[22629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513121, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 12:45:21.710309 22714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:21.700531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53796 (local address 127.0.98.62:42777)
0927 12:45:21.700689 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:21.700693 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:21.700703 (+ 10us) server_negotiation.cc:407] Connection header received
0927 12:45:21.700738 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:21.700740 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:21.700783 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:21.700878 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:21.701846 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:21.702335 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:21.703739 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:21.703956 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:21.704066 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:45:21.706558 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:45:21.706561 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:45:21.706563 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:45:21.706586 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:45:21.708180 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:45:21.708724 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:45:21.708728 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:45:21.708730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:45:21.708829 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:45:21.709166 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:45:21.709171 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:45:21.709173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:45:21.709357 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:45:21.709458 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 12:45:21.710022 (+ 564us) server_negotiation.cc:299] Negotiation successful
0927 12:45:21.710137 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 12:45:21.711138 22866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:21.699901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42777 (local address 127.0.0.1:53796)
0927 12:45:21.700257 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0927 12:45:21.700303 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 12:45:21.700421 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:45:21.700929 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:45:21.700938 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:45:21.701285 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:45:21.701696 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:45:21.701708 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:45:21.702443 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:45:21.702448 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:45:21.703613 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:45:21.703630 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:45:21.704094 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:45:21.704098 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:45:21.704226 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:45:21.704905 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:45:21.704919 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:45:21.706360 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:45:21.708292 (+ 1932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:45:21.708300 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:45:21.708308 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:45:21.708624 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:45:21.708931 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:45:21.708935 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:45:21.708938 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:45:21.709070 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:45:21.709467 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:45:21.709476 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:45:21.709781 (+ 305us) client_negotiation.cc:769] Sending connection context
0927 12:45:21.709954 (+ 173us) client_negotiation.cc:240] Negotiation successful
0927 12:45:21.710247 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":45,"spinlock_wait_cycles":768,"thread_start_us":128,"threads_started":1}
I20260927 12:45:21.711143 22865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:21.698196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34680 (local address 127.0.98.60:35603)
0927 12:45:21.698656 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:21.698674 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:21.698716 (+ 42us) server_negotiation.cc:407] Connection header received
0927 12:45:21.699535 (+ 819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:21.699557 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:21.699915 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:21.701146 (+ 1231us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:21.701486 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:21.702436 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:21.703574 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:21.703927 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:21.704094 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:45:21.706930 (+ 2836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:45:21.706954 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:45:21.706970 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:45:21.706996 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:45:21.708738 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:45:21.709141 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:45:21.709146 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:45:21.709148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:45:21.709228 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:45:21.709501 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:45:21.709505 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:45:21.709507 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:45:21.709822 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:45:21.710015 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0927 12:45:21.710241 (+ 226us) server_negotiation.cc:299] Negotiation successful
0927 12:45:21.710375 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":320,"thread_start_us":136,"threads_started":1}
I20260927 12:45:21.712944 22862 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43239 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 12:45:21 dist-test-slave-bnc1 krb5kdc[22629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513121, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 27 12:45:21 dist-test-slave-bnc1 krb5kdc[22629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513121, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:45:21.721553 22866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:21.713433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43239 (local address 127.0.0.1:55794)
0927 12:45:21.713533 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0927 12:45:21.713547 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:45:21.713638 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:45:21.713875 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:45:21.713879 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:45:21.714055 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:45:21.714151 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:45:21.714156 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:45:21.715066 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:45:21.715069 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:45:21.715642 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:45:21.715650 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:45:21.716173 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:45:21.716180 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:45:21.716286 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:45:21.716680 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:45:21.716690 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:45:21.718083 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:45:21.720023 (+ 1940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:45:21.720029 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:45:21.720032 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:45:21.720257 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:45:21.720529 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:45:21.720534 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:45:21.720538 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:45:21.720636 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:45:21.721048 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:45:21.721051 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:45:21.721146 (+ 95us) client_negotiation.cc:769] Sending connection context
0927 12:45:21.721251 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 12:45:21.721364 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260927 12:45:21.721630 22790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:21.713517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55794 (local address 127.0.98.61:43239)
0927 12:45:21.713649 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:21.713653 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:21.713695 (+ 42us) server_negotiation.cc:407] Connection header received
0927 12:45:21.713732 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:21.713734 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:21.713781 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:21.713839 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:21.714291 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:21.714964 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:21.715751 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:21.716066 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:21.716143 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:45:21.718217 (+ 2074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:45:21.718221 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:45:21.718224 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:45:21.718252 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:45:21.719910 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:45:21.720365 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:45:21.720368 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:45:21.720369 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:45:21.720443 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:45:21.720728 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:45:21.720734 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:45:21.720737 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:45:21.720954 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:45:21.721050 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 12:45:21.721330 (+ 280us) server_negotiation.cc:299] Negotiation successful
0927 12:45:21.721472 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260927 12:45:21.722335 22862 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35603 } has no permanent_uuid. Determining permanent_uuid...
I20260927 12:45:21.722693 22710 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35603 } attempt: 3
I20260927 12:45:21.722879 22714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:21.713408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53800 (local address 127.0.98.62:42777)
0927 12:45:21.713787 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:21.713790 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:21.713799 (+ 9us) server_negotiation.cc:407] Connection header received
0927 12:45:21.713847 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:21.713851 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:21.713892 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:21.713971 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:21.714621 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:21.715135 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:21.715898 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:21.716097 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:21.716214 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:45:21.718788 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:45:21.718792 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:45:21.718794 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:45:21.718825 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:45:21.720583 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:45:21.721048 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:45:21.721051 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:45:21.721053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:45:21.721133 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:45:21.721438 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:45:21.721441 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:45:21.721443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:45:21.721691 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:45:21.721945 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0927 12:45:21.722548 (+ 603us) server_negotiation.cc:299] Negotiation successful
0927 12:45:21.722715 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303}
I20260927 12:45:21.725507 22785 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35603 } attempt: 3
Sep 27 12:45:21 dist-test-slave-bnc1 krb5kdc[22629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513121, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 12:45:21 dist-test-slave-bnc1 krb5kdc[22629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513121, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 27 12:45:21 dist-test-slave-bnc1 krb5kdc[22629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513121, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 12:45:21 dist-test-slave-bnc1 krb5kdc[22629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513121, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 12:45:21.732869 22713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:21.723068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35603 (local address 127.0.0.1:34692)
0927 12:45:21.723438 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0927 12:45:21.723450 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0927 12:45:21.723555 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:45:21.723831 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:45:21.723833 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:45:21.724016 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:45:21.724111 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:45:21.724119 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:45:21.725053 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:45:21.725057 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:45:21.725815 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:45:21.725825 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:45:21.726455 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:45:21.726459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:45:21.726540 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:45:21.727020 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:45:21.727033 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:45:21.728656 (+ 1623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:45:21.731100 (+ 2444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:45:21.731103 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:45:21.731106 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:45:21.731363 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:45:21.731674 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:45:21.731678 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:45:21.731680 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:45:21.731768 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:45:21.732280 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:45:21.732285 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:45:21.732403 (+ 118us) client_negotiation.cc:769] Sending connection context
0927 12:45:21.732524 (+ 121us) client_negotiation.cc:240] Negotiation successful
0927 12:45:21.732667 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309}
I20260927 12:45:21.733532 22867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:21.723074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34692 (local address 127.0.98.60:35603)
0927 12:45:21.723378 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:21.723387 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:21.723518 (+ 131us) server_negotiation.cc:407] Connection header received
0927 12:45:21.723660 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:21.723663 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:21.723700 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:21.723805 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:21.724213 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:21.724954 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:21.725943 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:21.726324 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:21.726736 (+ 412us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:45:21.728798 (+ 2062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:45:21.728801 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:45:21.728804 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:45:21.728825 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:45:21.730958 (+ 2133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:45:21.731456 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:45:21.731463 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:45:21.731466 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:45:21.731553 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:45:21.731910 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:45:21.731916 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:45:21.731917 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:45:21.732129 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:45:21.732274 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 12:45:21.733136 (+ 862us) server_negotiation.cc:299] Negotiation successful
0927 12:45:21.733285 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":107,"threads_started":1}
I20260927 12:45:21.736099 22866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:21.722851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35603 (local address 127.0.0.1:34690)
0927 12:45:21.723702 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0927 12:45:21.723716 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:45:21.723798 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:45:21.724422 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:45:21.724425 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:45:21.724581 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:45:21.724654 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:45:21.724660 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:45:21.726129 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:45:21.726135 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:45:21.726884 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:45:21.726897 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:45:21.729674 (+ 2777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:45:21.729678 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:45:21.729799 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:45:21.730163 (+ 364us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:45:21.730182 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:45:21.731643 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:45:21.734490 (+ 2847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:45:21.734492 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:45:21.734494 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:45:21.734831 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:45:21.735109 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:45:21.735112 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:45:21.735114 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:45:21.735193 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:45:21.735575 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:45:21.735578 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:45:21.735663 (+ 85us) client_negotiation.cc:769] Sending connection context
0927 12:45:21.735784 (+ 121us) client_negotiation.cc:240] Negotiation successful
0927 12:45:21.735916 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":785,"mutex_wait_us":20}
I20260927 12:45:21.736099 22865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:21.723060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34690 (local address 127.0.98.60:35603)
0927 12:45:21.723280 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:21.723283 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:21.724227 (+ 944us) server_negotiation.cc:407] Connection header received
0927 12:45:21.724274 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:21.724279 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:21.724322 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:21.724390 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:21.724773 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:21.726019 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:21.729148 (+ 3129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:21.729525 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:21.729728 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:45:21.731779 (+ 2051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:45:21.731783 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:45:21.731787 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:45:21.731816 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:45:21.734380 (+ 2564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:45:21.734929 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:45:21.734935 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:45:21.734937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:45:21.735015 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:45:21.735282 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:45:21.735286 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:45:21.735288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:45:21.735472 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:45:21.735574 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 12:45:21.735828 (+ 254us) server_negotiation.cc:299] Negotiation successful
0927 12:45:21.735965 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":69,"server-negotiator.queue_time_us":171}
I20260927 12:45:21.737498 22787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:21.725884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35603 (local address 127.0.0.1:34708)
0927 12:45:21.726126 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0927 12:45:21.726139 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:45:21.726200 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:45:21.727608 (+ 1408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:45:21.727612 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:45:21.727798 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:45:21.727902 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:45:21.727909 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:45:21.729003 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:45:21.729009 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:45:21.729858 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:45:21.729870 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:45:21.730515 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:45:21.730519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:45:21.730609 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:45:21.731258 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:45:21.731281 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:45:21.732878 (+ 1597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:45:21.735996 (+ 3118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:45:21.736000 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:45:21.736002 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:45:21.736243 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:45:21.736560 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:45:21.736563 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:45:21.736565 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:45:21.736645 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:45:21.737061 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:45:21.737064 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:45:21.737152 (+ 88us) client_negotiation.cc:769] Sending connection context
0927 12:45:21.737225 (+ 73us) client_negotiation.cc:240] Negotiation successful
0927 12:45:21.737357 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169}
I20260927 12:45:21.737756 22710 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 109924a8231d4bfc9c6042b994a5434c: Bootstrap starting.
I20260927 12:45:21.738448 22868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:21.726322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34708 (local address 127.0.98.60:35603)
0927 12:45:21.727352 (+ 1030us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:21.727362 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:21.727373 (+ 11us) server_negotiation.cc:407] Connection header received
0927 12:45:21.727405 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:21.727411 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:21.727458 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:21.728020 (+ 562us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:21.728148 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:21.728904 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:21.729985 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:21.730372 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:21.731165 (+ 793us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:45:21.733276 (+ 2111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:45:21.733280 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:45:21.733285 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:45:21.733311 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:45:21.735874 (+ 2563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:45:21.736368 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:45:21.736371 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:45:21.736373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:45:21.736453 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:45:21.736753 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:45:21.736756 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:45:21.736757 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:45:21.736949 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:45:21.738117 (+ 1168us) server_negotiation.cc:1035] Waiting for connection context
0927 12:45:21.738189 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 12:45:21.738295 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":970,"thread_start_us":127,"threads_started":1}
I20260927 12:45:21.738862 22862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b4f6ce0f00747018d4095ba9b32a246: Bootstrap starting.
I20260927 12:45:21.740536 22710 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 109924a8231d4bfc9c6042b994a5434c: Neither blocks nor log segments found. Creating new log.
I20260927 12:45:21.741195 22710 log.cc:824] T 00000000000000000000000000000000 P 109924a8231d4bfc9c6042b994a5434c: Log is configured to *not* fsync() on all Append() calls
I20260927 12:45:21.741320 22862 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b4f6ce0f00747018d4095ba9b32a246: Neither blocks nor log segments found. Creating new log.
I20260927 12:45:21.741438 22785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73b52216271a44559be540b85a7a2f58: Bootstrap starting.
I20260927 12:45:21.742136 22862 log.cc:824] T 00000000000000000000000000000000 P 7b4f6ce0f00747018d4095ba9b32a246: Log is configured to *not* fsync() on all Append() calls
I20260927 12:45:21.742278 22790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:21.724180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55808 (local address 127.0.98.61:43239)
0927 12:45:21.724302 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:21.724306 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:21.724319 (+ 13us) server_negotiation.cc:407] Connection header received
0927 12:45:21.724466 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:21.724469 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:21.724508 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:21.724558 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:21.725415 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:21.726134 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:21.727178 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:21.727385 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:21.727506 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:45:21.730540 (+ 3034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:45:21.730545 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:45:21.730548 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:45:21.730572 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:45:21.733073 (+ 2501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:45:21.733734 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:45:21.733740 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:45:21.733747 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:45:21.733839 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:45:21.734154 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:45:21.734157 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:45:21.734159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:45:21.734334 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:45:21.734575 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0927 12:45:21.742070 (+ 7495us) server_negotiation.cc:299] Negotiation successful
0927 12:45:21.742173 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 12:45:21.743819 22785 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 73b52216271a44559be540b85a7a2f58: Neither blocks nor log segments found. Creating new log.
I20260927 12:45:21.743863 22710 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 109924a8231d4bfc9c6042b994a5434c: No bootstrap required, opened a new log
I20260927 12:45:21.744589 22785 log.cc:824] T 00000000000000000000000000000000 P 73b52216271a44559be540b85a7a2f58: Log is configured to *not* fsync() on all Append() calls
I20260927 12:45:21.746449 22862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b4f6ce0f00747018d4095ba9b32a246: No bootstrap required, opened a new log
I20260927 12:45:21.746613 22785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73b52216271a44559be540b85a7a2f58: No bootstrap required, opened a new log
I20260927 12:45:21.747807 22710 raft_consensus.cc:348] T 00000000000000000000000000000000 P 109924a8231d4bfc9c6042b994a5434c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "109924a8231d4bfc9c6042b994a5434c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42777 } } peers { permanent_uuid: "73b52216271a44559be540b85a7a2f58" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43239 } } peers { permanent_uuid: "7b4f6ce0f00747018d4095ba9b32a246" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35603 } }
I20260927 12:45:21.748035 22710 raft_consensus.cc:729] T 00000000000000000000000000000000 P 109924a8231d4bfc9c6042b994a5434c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 109924a8231d4bfc9c6042b994a5434c, State: Initialized, Role: FOLLOWER
I20260927 12:45:21.748389 22710 consensus_queue.cc:260] T 00000000000000000000000000000000 P 109924a8231d4bfc9c6042b994a5434c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "109924a8231d4bfc9c6042b994a5434c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42777 } } peers { permanent_uuid: "73b52216271a44559be540b85a7a2f58" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43239 } } peers { permanent_uuid: "7b4f6ce0f00747018d4095ba9b32a246" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35603 } }
I20260927 12:45:21.748917 22872 sys_catalog.cc:455] T 00000000000000000000000000000000 P 109924a8231d4bfc9c6042b994a5434c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "109924a8231d4bfc9c6042b994a5434c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42777 } } peers { permanent_uuid: "73b52216271a44559be540b85a7a2f58" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43239 } } peers { permanent_uuid: "7b4f6ce0f00747018d4095ba9b32a246" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35603 } } }
I20260927 12:45:21.749174 22872 sys_catalog.cc:458] T 00000000000000000000000000000000 P 109924a8231d4bfc9c6042b994a5434c [sys.catalog]: This master's current role is: FOLLOWER
W20260927 12:45:21.749513 22663 tablet.cc:2369] T 00000000000000000000000000000000 P 109924a8231d4bfc9c6042b994a5434c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 12:45:21.749758 22710 sys_catalog.cc:564] T 00000000000000000000000000000000 P 109924a8231d4bfc9c6042b994a5434c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:45:21.750365 22862 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b4f6ce0f00747018d4095ba9b32a246 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "109924a8231d4bfc9c6042b994a5434c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42777 } } peers { permanent_uuid: "73b52216271a44559be540b85a7a2f58" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43239 } } peers { permanent_uuid: "7b4f6ce0f00747018d4095ba9b32a246" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35603 } }
I20260927 12:45:21.750409 22785 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73b52216271a44559be540b85a7a2f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "109924a8231d4bfc9c6042b994a5434c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42777 } } peers { permanent_uuid: "73b52216271a44559be540b85a7a2f58" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43239 } } peers { permanent_uuid: "7b4f6ce0f00747018d4095ba9b32a246" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35603 } }
I20260927 12:45:21.750571 22862 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b4f6ce0f00747018d4095ba9b32a246 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b4f6ce0f00747018d4095ba9b32a246, State: Initialized, Role: FOLLOWER
I20260927 12:45:21.750666 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 12:45:21.750964 22862 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b4f6ce0f00747018d4095ba9b32a246 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "109924a8231d4bfc9c6042b994a5434c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42777 } } peers { permanent_uuid: "73b52216271a44559be540b85a7a2f58" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43239 } } peers { permanent_uuid: "7b4f6ce0f00747018d4095ba9b32a246" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35603 } }
I20260927 12:45:21.751364 22878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b4f6ce0f00747018d4095ba9b32a246 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "109924a8231d4bfc9c6042b994a5434c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42777 } } peers { permanent_uuid: "73b52216271a44559be540b85a7a2f58" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43239 } } peers { permanent_uuid: "7b4f6ce0f00747018d4095ba9b32a246" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35603 } } }
I20260927 12:45:21.751478 22878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b4f6ce0f00747018d4095ba9b32a246 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 12:45:21.751978 22785 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73b52216271a44559be540b85a7a2f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73b52216271a44559be540b85a7a2f58, State: Initialized, Role: FOLLOWER
I20260927 12:45:21.752213 22862 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b4f6ce0f00747018d4095ba9b32a246 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:45:21.752291 22785 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73b52216271a44559be540b85a7a2f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "109924a8231d4bfc9c6042b994a5434c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42777 } } peers { permanent_uuid: "73b52216271a44559be540b85a7a2f58" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43239 } } peers { permanent_uuid: "7b4f6ce0f00747018d4095ba9b32a246" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35603 } }
I20260927 12:45:21.752213 22684 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "109924a8231d4bfc9c6042b994a5434c"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53800
I20260927 12:45:21.752736 22684 raft_consensus.cc:482] T 00000000000000000000000000000000 P 109924a8231d4bfc9c6042b994a5434c [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 12:45:21.752839 22684 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 109924a8231d4bfc9c6042b994a5434c [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:45:21.753525 22785 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73b52216271a44559be540b85a7a2f58 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:45:21.754156 22684 raft_consensus.cc:504] T 00000000000000000000000000000000 P 109924a8231d4bfc9c6042b994a5434c [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "109924a8231d4bfc9c6042b994a5434c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42777 } } peers { permanent_uuid: "73b52216271a44559be540b85a7a2f58" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43239 } } peers { permanent_uuid: "7b4f6ce0f00747018d4095ba9b32a246" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35603 } }
I20260927 12:45:21.755128 22684 leader_election.cc:288] T 00000000000000000000000000000000 P 109924a8231d4bfc9c6042b994a5434c [CANDIDATE]: Term 1 election: Requested vote from peers 73b52216271a44559be540b85a7a2f58 (127.0.98.61:43239), 7b4f6ce0f00747018d4095ba9b32a246 (127.0.98.60:35603)
W20260927 12:45:21.755647 22884 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 109924a8231d4bfc9c6042b994a5434c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 12:45:21.755712 22884 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 12:45:21.756366 22759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "109924a8231d4bfc9c6042b994a5434c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "73b52216271a44559be540b85a7a2f58"
I20260927 12:45:21.756597 22759 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73b52216271a44559be540b85a7a2f58 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:45:21.757440 22836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "109924a8231d4bfc9c6042b994a5434c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7b4f6ce0f00747018d4095ba9b32a246"
I20260927 12:45:21.758522 22836 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b4f6ce0f00747018d4095ba9b32a246 [term 0 FOLLOWER]: Advancing to term 1
W20260927 12:45:21.759783 22897 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7b4f6ce0f00747018d4095ba9b32a246: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 12:45:21.759845 22897 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 12:45:21.759865 22836 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7b4f6ce0f00747018d4095ba9b32a246 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 109924a8231d4bfc9c6042b994a5434c in term 1.
I20260927 12:45:21.760450 22659 leader_election.cc:302] T 00000000000000000000000000000000 P 109924a8231d4bfc9c6042b994a5434c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 109924a8231d4bfc9c6042b994a5434c, 7b4f6ce0f00747018d4095ba9b32a246; no voters:
I20260927 12:45:21.760728 22872 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 109924a8231d4bfc9c6042b994a5434c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:45:21.760962 22872 raft_consensus.cc:686] T 00000000000000000000000000000000 P 109924a8231d4bfc9c6042b994a5434c [term 1 LEADER]: Becoming Leader. State: Replica: 109924a8231d4bfc9c6042b994a5434c, State: Running, Role: LEADER
I20260927 12:45:21.761289 22872 consensus_queue.cc:237] T 00000000000000000000000000000000 P 109924a8231d4bfc9c6042b994a5434c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "109924a8231d4bfc9c6042b994a5434c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42777 } } peers { permanent_uuid: "73b52216271a44559be540b85a7a2f58" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43239 } } peers { permanent_uuid: "7b4f6ce0f00747018d4095ba9b32a246" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35603 } }
I20260927 12:45:21.757943 22759 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 73b52216271a44559be540b85a7a2f58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 109924a8231d4bfc9c6042b994a5434c in term 1.
I20260927 12:45:21.765957 22792 master_runner.cc:263] No leader master found from master 7b4f6ce0f00747018d4095ba9b32a246
I20260927 12:45:21.765898 22914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 109924a8231d4bfc9c6042b994a5434c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 109924a8231d4bfc9c6042b994a5434c. Latest consensus state: current_term: 1 leader_uuid: "109924a8231d4bfc9c6042b994a5434c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "109924a8231d4bfc9c6042b994a5434c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42777 } } peers { permanent_uuid: "73b52216271a44559be540b85a7a2f58" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43239 } } peers { permanent_uuid: "7b4f6ce0f00747018d4095ba9b32a246" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35603 } } }
I20260927 12:45:21.766017 22914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 109924a8231d4bfc9c6042b994a5434c [sys.catalog]: This master's current role is: LEADER
I20260927 12:45:21.766026 22792 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 12:45:21.766426 22917 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260927 12:45:21.767992 22909 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 73b52216271a44559be540b85a7a2f58: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 12:45:21.768081 22909 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 12:45:21.769049 22885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73b52216271a44559be540b85a7a2f58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "109924a8231d4bfc9c6042b994a5434c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42777 } } peers { permanent_uuid: "73b52216271a44559be540b85a7a2f58" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43239 } } peers { permanent_uuid: "7b4f6ce0f00747018d4095ba9b32a246" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35603 } } }
I20260927 12:45:21.769212 22885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73b52216271a44559be540b85a7a2f58 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 12:45:21.770035 22717 master_runner.cc:263] No leader master found from master 73b52216271a44559be540b85a7a2f58
I20260927 12:45:21.770095 22717 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 12:45:21.771946 22642 master_runner.cc:263] No leader master found from master 109924a8231d4bfc9c6042b994a5434c
I20260927 12:45:21.772006 22642 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 12:45:21.772974 22917 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:45:21.779076 22836 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7b4f6ce0f00747018d4095ba9b32a246 [term 1 FOLLOWER]: Refusing update from remote peer 109924a8231d4bfc9c6042b994a5434c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:45:21.779820 22914 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 109924a8231d4bfc9c6042b994a5434c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b4f6ce0f00747018d4095ba9b32a246" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:45:21.783818 22878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b4f6ce0f00747018d4095ba9b32a246 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 109924a8231d4bfc9c6042b994a5434c. Latest consensus state: current_term: 1 leader_uuid: "109924a8231d4bfc9c6042b994a5434c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "109924a8231d4bfc9c6042b994a5434c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42777 } } peers { permanent_uuid: "73b52216271a44559be540b85a7a2f58" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43239 } } peers { permanent_uuid: "7b4f6ce0f00747018d4095ba9b32a246" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35603 } } }
I20260927 12:45:21.783943 22878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b4f6ce0f00747018d4095ba9b32a246 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 12:45:21.785066 22759 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 73b52216271a44559be540b85a7a2f58 [term 1 FOLLOWER]: Refusing update from remote peer 109924a8231d4bfc9c6042b994a5434c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:45:21.785734 22914 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 109924a8231d4bfc9c6042b994a5434c [LEADER]: Connected to new peer: Peer: permanent_uuid: "73b52216271a44559be540b85a7a2f58" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:45:21.789276 22885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73b52216271a44559be540b85a7a2f58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 109924a8231d4bfc9c6042b994a5434c. Latest consensus state: current_term: 1 leader_uuid: "109924a8231d4bfc9c6042b994a5434c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "109924a8231d4bfc9c6042b994a5434c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42777 } } peers { permanent_uuid: "73b52216271a44559be540b85a7a2f58" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43239 } } peers { permanent_uuid: "7b4f6ce0f00747018d4095ba9b32a246" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35603 } } }
I20260927 12:45:21.789390 22885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73b52216271a44559be540b85a7a2f58 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 12:45:21.794852 22914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 109924a8231d4bfc9c6042b994a5434c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "109924a8231d4bfc9c6042b994a5434c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "109924a8231d4bfc9c6042b994a5434c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42777 } } peers { permanent_uuid: "73b52216271a44559be540b85a7a2f58" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43239 } } peers { permanent_uuid: "7b4f6ce0f00747018d4095ba9b32a246" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35603 } } }
I20260927 12:45:21.794967 22914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 109924a8231d4bfc9c6042b994a5434c [sys.catalog]: This master's current role is: LEADER
I20260927 12:45:21.796219 22885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73b52216271a44559be540b85a7a2f58 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "109924a8231d4bfc9c6042b994a5434c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "109924a8231d4bfc9c6042b994a5434c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42777 } } peers { permanent_uuid: "73b52216271a44559be540b85a7a2f58" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43239 } } peers { permanent_uuid: "7b4f6ce0f00747018d4095ba9b32a246" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35603 } } }
I20260927 12:45:21.796331 22885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73b52216271a44559be540b85a7a2f58 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 12:45:21.796738 22921 mvcc.cc:205] Tried to move back new op lower bound from 7333941746798481408 to 7333941746737807360. Current Snapshot: MvccSnapshot[applied={T|T < 7333941746798481408}]
I20260927 12:45:21.798460 22714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:21.768573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53816 (local address 127.0.98.62:42777)
0927 12:45:21.768695 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:21.768700 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:21.777883 (+ 9183us) server_negotiation.cc:407] Connection header received
0927 12:45:21.778092 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:21.778095 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:21.778143 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:21.778193 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:21.779266 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:21.780055 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:21.784294 (+ 4239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:21.784648 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:21.784722 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:45:21.792846 (+ 8124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:45:21.792850 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:45:21.792853 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:45:21.792882 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:45:21.794613 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:45:21.797034 (+ 2421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:45:21.797038 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:45:21.797040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:45:21.797128 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:45:21.797421 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:45:21.797424 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:45:21.797427 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:45:21.797670 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:45:21.797832 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 12:45:21.798179 (+ 347us) server_negotiation.cc:299] Negotiation successful
0927 12:45:21.798308 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260927 12:45:21.798823 22924 mvcc.cc:205] Tried to move back new op lower bound from 7333941746798481408 to 7333941746737807360. Current Snapshot: MvccSnapshot[applied={T|T < 7333941746798481408}]
I20260927 12:45:21.799525 22872 sys_catalog.cc:455] T 00000000000000000000000000000000 P 109924a8231d4bfc9c6042b994a5434c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "109924a8231d4bfc9c6042b994a5434c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "109924a8231d4bfc9c6042b994a5434c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42777 } } peers { permanent_uuid: "73b52216271a44559be540b85a7a2f58" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43239 } } peers { permanent_uuid: "7b4f6ce0f00747018d4095ba9b32a246" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35603 } } }
I20260927 12:45:21.799635 22872 sys_catalog.cc:458] T 00000000000000000000000000000000 P 109924a8231d4bfc9c6042b994a5434c [sys.catalog]: This master's current role is: LEADER
I20260927 12:45:21.800675 22878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b4f6ce0f00747018d4095ba9b32a246 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "109924a8231d4bfc9c6042b994a5434c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "109924a8231d4bfc9c6042b994a5434c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42777 } } peers { permanent_uuid: "73b52216271a44559be540b85a7a2f58" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43239 } } peers { permanent_uuid: "7b4f6ce0f00747018d4095ba9b32a246" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35603 } } }
I20260927 12:45:21.800793 22878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b4f6ce0f00747018d4095ba9b32a246 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 12:45:21.804075 22922 mvcc.cc:205] Tried to move back new op lower bound from 7333941746798481408 to 7333941746737807360. Current Snapshot: MvccSnapshot[applied={T|T < 7333941746798481408}]
I20260927 12:45:21.806506 22917 catalog_manager.cc:1347] Generated new cluster ID: 0ab6ae9626ac4743a78288a45c9196b0
I20260927 12:45:21.806564 22917 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:45:21.806938 22868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:21.770152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34720 (local address 127.0.98.60:35603)
0927 12:45:21.770264 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:21.770270 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:21.781851 (+ 11581us) server_negotiation.cc:407] Connection header received
0927 12:45:21.783269 (+ 1418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:21.783273 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:21.783320 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:21.783370 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:21.787375 (+ 4005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:21.788178 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:21.791628 (+ 3450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:21.792019 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:21.792099 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:45:21.801925 (+ 9826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:45:21.801929 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:45:21.801932 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:45:21.801961 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:45:21.803635 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:45:21.804526 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:45:21.804530 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:45:21.804532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:45:21.804620 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:45:21.805305 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:45:21.805309 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:45:21.805311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:45:21.805530 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:45:21.805701 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 12:45:21.806641 (+ 940us) server_negotiation.cc:299] Negotiation successful
0927 12:45:21.806784 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 12:45:21.808943 22790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:21.761974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55820 (local address 127.0.98.61:43239)
0927 12:45:21.762107 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:21.762111 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:21.781604 (+ 19493us) server_negotiation.cc:407] Connection header received
0927 12:45:21.782036 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:21.782040 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:21.782096 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:21.782148 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:21.785449 (+ 3301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:21.786242 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:21.790567 (+ 4325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:21.790949 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:21.791019 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:45:21.800015 (+ 8996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:45:21.800020 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:45:21.800023 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:45:21.800050 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:45:21.801765 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:45:21.804932 (+ 3167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:45:21.804935 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:45:21.804937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:45:21.805018 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:45:21.806024 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:45:21.806028 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:45:21.806030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:45:21.806230 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:45:21.806336 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 12:45:21.808655 (+ 2319us) server_negotiation.cc:299] Negotiation successful
0927 12:45:21.808791 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 12:45:21.826076 22917 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:45:21.827510 22917 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:45:21.846099 22917 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 109924a8231d4bfc9c6042b994a5434c: Generated new TSK 0
I20260927 12:45:21.847499 22917 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 12:45:21.881925 22714 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:45:21.878869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53828 (local address 127.0.98.62:42777)
0927 12:45:21.879001 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:21.879005 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:21.880795 (+ 1790us) server_negotiation.cc:407] Connection header received
0927 12:45:21.880993 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:21.880997 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:21.881061 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:21.881117 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:21.881791 (+ 674us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53828: BlockingRecv error: recv got EOF from 127.0.0.1:53828 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260927 12:45:21.883505 22790 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:45:21.879696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55828 (local address 127.0.98.61:43239)
0927 12:45:21.879812 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:21.879817 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:21.882646 (+ 2829us) server_negotiation.cc:407] Connection header received
0927 12:45:21.882835 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:21.882838 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:21.882884 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:21.882934 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:21.883395 (+ 461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55828: BlockingRecv error: recv got EOF from 127.0.0.1:55828 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260927 12:45:21.884922 22868 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:45:21.880291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34728 (local address 127.0.98.60:35603)
0927 12:45:21.880444 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:21.880450 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:21.883981 (+ 3531us) server_negotiation.cc:407] Connection header received
0927 12:45:21.884270 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:21.884273 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:21.884318 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:21.884366 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:21.884813 (+ 447us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34728: BlockingRecv error: recv got EOF from 127.0.0.1:34728 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260927 12:45:21.889356 22790 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:45:21.887912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55830 (local address 127.0.98.61:43239)
0927 12:45:21.888009 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:21.888013 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:21.888447 (+ 434us) server_negotiation.cc:407] Connection header received
0927 12:45:21.888655 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:21.888660 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:21.888747 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:21.888796 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:21.889263 (+ 467us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55830: BlockingRecv error: recv got EOF from 127.0.0.1:55830 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260927 12:45:21.894551 22790 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:45:21.893011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55842 (local address 127.0.98.61:43239)
0927 12:45:21.893107 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:21.893110 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:21.893674 (+ 564us) server_negotiation.cc:407] Connection header received
0927 12:45:21.893865 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:21.893868 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:21.893917 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:21.893968 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:21.894452 (+ 484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55842: BlockingRecv error: recv got EOF from 127.0.0.1:55842 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260927 12:45:21.900302 22790 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:45:21.898838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55854 (local address 127.0.98.61:43239)
0927 12:45:21.898938 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:21.898942 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:21.899450 (+ 508us) server_negotiation.cc:407] Connection header received
0927 12:45:21.899619 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:21.899622 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:21.899682 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:21.899729 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:21.900199 (+ 470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55854: BlockingRecv error: recv got EOF from 127.0.0.1:55854 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260927 12:45:21.908294 22790 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:45:21.906809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55868 (local address 127.0.98.61:43239)
0927 12:45:21.906950 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:21.906955 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:21.907433 (+ 478us) server_negotiation.cc:407] Connection header received
0927 12:45:21.907614 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:21.907617 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:21.907675 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:21.907724 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:21.908197 (+ 473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55868: BlockingRecv error: recv got EOF from 127.0.0.1:55868 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260927 12:45:21.920449 22790 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:45:21.918759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55882 (local address 127.0.98.61:43239)
0927 12:45:21.918893 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:21.918898 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:21.919434 (+ 536us) server_negotiation.cc:407] Connection header received
0927 12:45:21.919638 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:21.919641 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:21.919711 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:21.919770 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:21.920341 (+ 571us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55882: BlockingRecv error: recv got EOF from 127.0.0.1:55882 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260927 12:45:21.941098 22790 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:45:21.939786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55898 (local address 127.0.98.61:43239)
0927 12:45:21.939887 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:21.939892 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:21.940017 (+ 125us) server_negotiation.cc:407] Connection header received
0927 12:45:21.940213 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:21.940216 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:21.940265 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:21.940333 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:21.940978 (+ 645us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55898: BlockingRecv error: recv got EOF from 127.0.0.1:55898 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260927 12:45:21.977044 22790 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:45:21.975597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55912 (local address 127.0.98.61:43239)
0927 12:45:21.975833 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:21.975838 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:21.976037 (+ 199us) server_negotiation.cc:407] Connection header received
0927 12:45:21.976250 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:21.976254 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:21.976312 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:21.976373 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:21.976913 (+ 540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55912: BlockingRecv error: recv got EOF from 127.0.0.1:55912 (error 108)
Metrics: {"server-negotiator.queue_time_us":115}
W20260927 12:45:22.044276 22790 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:45:22.043174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55922 (local address 127.0.98.61:43239)
0927 12:45:22.043368 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:22.043372 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:22.043456 (+ 84us) server_negotiation.cc:407] Connection header received
0927 12:45:22.043579 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:22.043582 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:22.043630 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:22.043705 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:22.044146 (+ 441us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55922: BlockingRecv error: recv got EOF from 127.0.0.1:55922 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260927 12:45:22.175604 22790 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:45:22.174393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55926 (local address 127.0.98.61:43239)
0927 12:45:22.174560 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:22.174564 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:22.174729 (+ 165us) server_negotiation.cc:407] Connection header received
0927 12:45:22.174899 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:22.174902 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:22.174958 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:22.175049 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:22.175496 (+ 447us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55926: BlockingRecv error: recv got EOF from 127.0.0.1:55926 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260927 12:45:22.434952 22790 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:45:22.433806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55940 (local address 127.0.98.61:43239)
0927 12:45:22.433938 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:22.433942 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:22.434110 (+ 168us) server_negotiation.cc:407] Connection header received
0927 12:45:22.434269 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:22.434271 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:22.434317 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:22.434388 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:22.434826 (+ 438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55940: BlockingRecv error: recv got EOF from 127.0.0.1:55940 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 12:45:22.761112 22897 catalog_manager.cc:1259] Loaded cluster ID: 0ab6ae9626ac4743a78288a45c9196b0
I20260927 12:45:22.761211 22897 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7b4f6ce0f00747018d4095ba9b32a246: loading cluster ID for follower catalog manager: success
I20260927 12:45:22.762969 22897 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7b4f6ce0f00747018d4095ba9b32a246: acquiring CA information for follower catalog manager: success
I20260927 12:45:22.764108 22897 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7b4f6ce0f00747018d4095ba9b32a246: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 12:45:22.769181 22909 catalog_manager.cc:1259] Loaded cluster ID: 0ab6ae9626ac4743a78288a45c9196b0
I20260927 12:45:22.769243 22909 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 73b52216271a44559be540b85a7a2f58: loading cluster ID for follower catalog manager: success
I20260927 12:45:22.770852 22909 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 73b52216271a44559be540b85a7a2f58: acquiring CA information for follower catalog manager: success
I20260927 12:45:22.771911 22909 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 73b52216271a44559be540b85a7a2f58: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 12:45:22.953346 22993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:22.949008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55948 (local address 127.0.98.61:43239)
0927 12:45:22.949260 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:22.949264 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:22.949317 (+ 53us) server_negotiation.cc:407] Connection header received
0927 12:45:22.949441 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:22.949443 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:22.949606 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:22.949722 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:45:22.950002 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:22.950588 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:22.951506 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:22.951819 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:22.951962 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:45:22.952321 (+ 359us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:45:22.952844 (+ 523us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:45:22.952939 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 12:45:22.953087 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 12:45:22.953152 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":80,"threads_started":1}
I20260927 12:45:22.960228 22999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:22.956525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34736 (local address 127.0.98.60:35603)
0927 12:45:22.956777 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:22.956784 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:22.956801 (+ 17us) server_negotiation.cc:407] Connection header received
0927 12:45:22.956892 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:22.956894 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:22.957004 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:22.957125 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:45:22.957344 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:22.957912 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:22.958747 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:22.958979 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:22.959089 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:45:22.959392 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:45:22.959833 (+ 441us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:45:22.959974 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 12:45:22.960009 (+ 35us) server_negotiation.cc:299] Negotiation successful
0927 12:45:22.960064 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":73,"threads_started":1}
I20260927 12:45:22.967540 22993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:22.963415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55952 (local address 127.0.98.61:43239)
0927 12:45:22.963538 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:22.963542 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:22.963946 (+ 404us) server_negotiation.cc:407] Connection header received
0927 12:45:22.964156 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:22.964161 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:22.964233 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:22.964273 (+ 40us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:45:22.964609 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:22.965275 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:22.966374 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:22.966711 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:22.966814 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:45:22.967055 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:45:22.967144 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:45:22.967249 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 12:45:22.967341 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 12:45:22.967395 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 12:45:22.968174 23006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:22.963160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53832 (local address 127.0.98.62:42777)
0927 12:45:22.963412 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:22.963416 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:22.963432 (+ 16us) server_negotiation.cc:407] Connection header received
0927 12:45:22.963557 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:22.963561 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:22.963758 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:22.963891 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:45:22.964116 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:22.964946 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:22.966155 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:22.966484 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:22.966660 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:45:22.967000 (+ 340us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:45:22.967359 (+ 359us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:45:22.967946 (+ 587us) server_negotiation.cc:1035] Waiting for connection context
0927 12:45:22.967981 (+ 35us) server_negotiation.cc:299] Negotiation successful
0927 12:45:22.968040 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":101,"threads_started":1}
I20260927 12:45:22.968320 22999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:22.963588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34740 (local address 127.0.98.60:35603)
0927 12:45:22.963693 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:22.963697 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:22.963942 (+ 245us) server_negotiation.cc:407] Connection header received
0927 12:45:22.964211 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:22.964220 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:22.964271 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:22.964344 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:45:22.965408 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:22.966008 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:22.967128 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:22.967484 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:22.967585 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:45:22.967868 (+ 283us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:45:22.967945 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:45:22.967993 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 12:45:22.968138 (+ 145us) server_negotiation.cc:299] Negotiation successful
0927 12:45:22.968187 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 12:45:22.971614 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 22642
I20260927 12:45:22.978344 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 22717
I20260927 12:45:22.984098 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 22792
2026-09-27T12:45:22Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3085 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512840836255-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 12:45:23 dist-test-slave-bnc1 krb5kdc[23010](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 12:45:23 dist-test-slave-bnc1 krb5kdc[23010](info): set up 2 sockets
Sep 27 12:45:23 dist-test-slave-bnc1 krb5kdc[23010](info): commencing operation
krb5kdc: starting...
W20260927 12:45:24.042152 392 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512840836255-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512840836255-392-0/krb5kdc/test-user.keytab.
Sep 27 12:45:24 dist-test-slave-bnc1 krb5kdc[23010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T12:45:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T12:45:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260927 12:45:24.162336 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512840836255-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512840836255-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512840836255-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33161
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33161,127.0.98.61:43945,127.0.98.60:36441
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 12:45:24.271456 23023 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 12:45:24.271778 23023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:45:24.271852 23023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:45:24.271934 23023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:45:24.275851 23023 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:45:24.275931 23023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:45:24.275970 23023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:45:24.276007 23023 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:45:24.276042 23023 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:45:24.281251 23023 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512840836255-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512840836255-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33161,127.0.98.61:43945,127.0.98.60:36441
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512840836255-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23023
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512840836255-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:45:24.282267 23023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:45:24.283241 23023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:45:24.291673 23028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:45:24.291685 23029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:45:24.291927 23031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:45:24.291987 23023 server_base.cc:1034] running on GCE node
I20260927 12:45:24.292572 23023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:45:24.293764 23023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:45:24.294943 23023 hybrid_clock.cc:648] HybridClock initialized: now 1790513124294924 us; error 33 us; skew 500 ppm
Sep 27 12:45:24 dist-test-slave-bnc1 krb5kdc[23010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513124, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:45:24.297652 23023 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260927 12:45:24.298452 23023 webserver.cc:460] Webserver started at http://127.0.98.62:36899/ using document root <none> and password file <none>
I20260927 12:45:24.298967 23023 fs_manager.cc:359] Metadata directory not provided
I20260927 12:45:24.299022 23023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:45:24.299189 23023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:45:24.301115 23023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512840836255-392-0/minicluster-data/master-0/data/instance:
uuid: "c3add08c79334ab2895224604e7531e3"
format_stamp: "Formatted at 2026-09-27 12:45:24 on dist-test-slave-bnc1"
server_key: "da1d4f75eb51db615c8861165a32c1a2"
server_key_iv: "389beb2ea9cd54b6fbaf76c77b54a87d"
server_key_version: "encryptionkey@0"
I20260927 12:45:24.301684 23023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512840836255-392-0/minicluster-data/master-0/wal/instance:
uuid: "c3add08c79334ab2895224604e7531e3"
format_stamp: "Formatted at 2026-09-27 12:45:24 on dist-test-slave-bnc1"
server_key: "da1d4f75eb51db615c8861165a32c1a2"
server_key_iv: "389beb2ea9cd54b6fbaf76c77b54a87d"
server_key_version: "encryptionkey@0"
I20260927 12:45:24.305109 23023 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 12:45:24.307382 23038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:45:24.308544 23023 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:45:24.308667 23023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512840836255-392-0/minicluster-data/master-0/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512840836255-392-0/minicluster-data/master-0/wal
uuid: "c3add08c79334ab2895224604e7531e3"
format_stamp: "Formatted at 2026-09-27 12:45:24 on dist-test-slave-bnc1"
server_key: "da1d4f75eb51db615c8861165a32c1a2"
server_key_iv: "389beb2ea9cd54b6fbaf76c77b54a87d"
server_key_version: "encryptionkey@0"
I20260927 12:45:24.308768 23023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512840836255-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512840836255-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512840836255-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:45:24.317540 23023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:45:24.320199 23023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:45:24.320405 23023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:45:24.326996 23023 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33161
I20260927 12:45:24.327291 23090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33161 every 8 connection(s)
I20260927 12:45:24.327934 23023 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512840836255-392-0/minicluster-data/master-0/data/info.pb
I20260927 12:45:24.330585 23091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:45:24.334240 23091 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33161 } has no permanent_uuid. Determining permanent_uuid...
I20260927 12:45:24.338131 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 23023
I20260927 12:45:24.338277 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512840836255-392-0/minicluster-data/master-0/wal/instance
I20260927 12:45:24.338611 392 external_mini_cluster.cc:1428] Setting key f037655fc17bf14b76a24b3c7018eb88
Sep 27 12:45:24 dist-test-slave-bnc1 krb5kdc[23010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513124, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
I20260927 12:45:24.349977 23093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:24.336322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33161 (local address 127.0.0.1:58146)
0927 12:45:24.336777 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0927 12:45:24.336825 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 12:45:24.337719 (+ 894us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:45:24.338554 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:45:24.338559 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:45:24.338956 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:45:24.339204 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:45:24.339217 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:45:24.340728 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:45:24.340735 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:45:24.342017 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:45:24.342031 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:45:24.342677 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:45:24.342683 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:45:24.342851 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:45:24.343426 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:45:24.343450 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:45:24.345267 (+ 1817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:45:24.347278 (+ 2011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:45:24.347286 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:45:24.347293 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:45:24.347503 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:45:24.347880 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:45:24.347882 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:45:24.347884 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:45:24.347986 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:45:24.348521 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:45:24.348526 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:45:24.348642 (+ 116us) client_negotiation.cc:769] Sending connection context
0927 12:45:24.348784 (+ 142us) client_negotiation.cc:240] Negotiation successful
0927 12:45:24.349056 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":1,"thread_start_us":144,"threads_started":1}
I20260927 12:45:24.349982 23094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:24.336322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58146 (local address 127.0.98.62:33161)
0927 12:45:24.336735 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:24.336750 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:24.336931 (+ 181us) server_negotiation.cc:407] Connection header received
0927 12:45:24.338055 (+ 1124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:24.338062 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:24.338346 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:24.338560 (+ 214us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:24.339357 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:24.340593 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:24.342151 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:24.342506 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:24.342690 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:45:24.345417 (+ 2727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:45:24.345422 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:45:24.345431 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:45:24.345451 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:45:24.347179 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:45:24.347598 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:45:24.347602 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:45:24.347603 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:45:24.347780 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:45:24.348067 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:45:24.348069 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:45:24.348070 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:45:24.348424 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:45:24.348583 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 12:45:24.348857 (+ 274us) server_negotiation.cc:299] Negotiation successful
0927 12:45:24.349163 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":100,"server-negotiator.queue_time_us":286,"spinlock_wait_cycles":59776,"thread_start_us":157,"threads_started":1}
I20260927 12:45:24.352324 23091 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43945 } has no permanent_uuid. Determining permanent_uuid...
I20260927 12:45:24.353127 23093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:24.352800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43945 (local address 127.0.0.1:33274)
0927 12:45:24.352951 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 12:45:24.353042 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260927 12:45:24.353330 23040 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43945: connect: Connection refused (error 111)
I20260927 12:45:24.354295 23093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:24.354002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43945 (local address 127.0.0.1:33280)
0927 12:45:24.354141 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0927 12:45:24.354214 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 12:45:24.354522 23091 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43945: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43945: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512840836255-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512840836255-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260927 12:45:24.366178 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512840836255-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512840836255-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512840836255-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512840836255-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43945
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512840836255-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33161,127.0.98.61:43945,127.0.98.60:36441
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 12:45:24.403789 23091 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43945 } attempt: 1
I20260927 12:45:24.404737 23093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:24.404382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43945 (local address 127.0.0.1:33288)
0927 12:45:24.404565 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0927 12:45:24.404642 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260927 12:45:24.405619 23093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:24.405361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43945 (local address 127.0.0.1:33292)
0927 12:45:24.405474 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0927 12:45:24.405526 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 12:45:24.405776 23091 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43945: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43945: connect: Connection refused (error 111)
I20260927 12:45:24.473960 23091 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43945 } attempt: 2
I20260927 12:45:24.474881 23093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:24.474583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43945 (local address 127.0.0.1:33298)
0927 12:45:24.474745 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 12:45:24.474807 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260927 12:45:24.475594 23093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:24.475377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43945 (local address 127.0.0.1:33312)
0927 12:45:24.475488 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0927 12:45:24.475540 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 12:45:24.475728 23091 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43945: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43945: connect: Connection refused (error 111)
W20260927 12:45:24.477782 23098 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 12:45:24.478104 23098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:45:24.478163 23098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:45:24.478241 23098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:45:24.482476 23098 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:45:24.482563 23098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:45:24.482612 23098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:45:24.482658 23098 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:45:24.482713 23098 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:45:24.488157 23098 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512840836255-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512840836255-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33161,127.0.98.61:43945,127.0.98.60:36441
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512840836255-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512840836255-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23098
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512840836255-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:45:24.489283 23098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:45:24.490379 23098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:45:24.498755 23104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:45:24.498740 23103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:45:24.498768 23106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:45:24.498857 23098 server_base.cc:1034] running on GCE node
I20260927 12:45:24.499521 23098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:45:24.500721 23098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:45:24.501905 23098 hybrid_clock.cc:648] HybridClock initialized: now 1790513124501891 us; error 28 us; skew 500 ppm
Sep 27 12:45:24 dist-test-slave-bnc1 krb5kdc[23010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513124, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:45:24.504582 23098 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260927 12:45:24.505519 23098 webserver.cc:460] Webserver started at http://127.0.98.61:42663/ using document root <none> and password file <none>
I20260927 12:45:24.506165 23098 fs_manager.cc:359] Metadata directory not provided
I20260927 12:45:24.506256 23098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:45:24.506491 23098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:45:24.508610 23098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512840836255-392-0/minicluster-data/master-1/data/instance:
uuid: "7f326d8addfb42388cc16d0f9559eb93"
format_stamp: "Formatted at 2026-09-27 12:45:24 on dist-test-slave-bnc1"
server_key: "a2e3931debb5d22c222d6590fbf60532"
server_key_iv: "c466b4e309775f293330293a161290fb"
server_key_version: "encryptionkey@0"
I20260927 12:45:24.509208 23098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512840836255-392-0/minicluster-data/master-1/wal/instance:
uuid: "7f326d8addfb42388cc16d0f9559eb93"
format_stamp: "Formatted at 2026-09-27 12:45:24 on dist-test-slave-bnc1"
server_key: "a2e3931debb5d22c222d6590fbf60532"
server_key_iv: "c466b4e309775f293330293a161290fb"
server_key_version: "encryptionkey@0"
I20260927 12:45:24.513271 23098 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260927 12:45:24.515805 23113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:45:24.517072 23098 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 12:45:24.517241 23098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512840836255-392-0/minicluster-data/master-1/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512840836255-392-0/minicluster-data/master-1/wal
uuid: "7f326d8addfb42388cc16d0f9559eb93"
format_stamp: "Formatted at 2026-09-27 12:45:24 on dist-test-slave-bnc1"
server_key: "a2e3931debb5d22c222d6590fbf60532"
server_key_iv: "c466b4e309775f293330293a161290fb"
server_key_version: "encryptionkey@0"
I20260927 12:45:24.517366 23098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512840836255-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512840836255-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512840836255-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:45:24.531292 23098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:45:24.534355 23098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:45:24.534581 23098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:45:24.541642 23098 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:43945
I20260927 12:45:24.541941 23165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:43945 every 8 connection(s)
I20260927 12:45:24.542743 23098 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512840836255-392-0/minicluster-data/master-1/data/info.pb
I20260927 12:45:24.545656 23166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:45:24.549376 23166 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33161 } has no permanent_uuid. Determining permanent_uuid...
I20260927 12:45:24.552506 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 23098
I20260927 12:45:24.552654 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512840836255-392-0/minicluster-data/master-1/wal/instance
I20260927 12:45:24.552953 392 external_mini_cluster.cc:1428] Setting key 88c9b937c19ff80608074fbad1dc2f18
Sep 27 12:45:24 dist-test-slave-bnc1 krb5kdc[23010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513124, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260927 12:45:24.564095 23094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:24.551368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58162 (local address 127.0.98.62:33161)
0927 12:45:24.551508 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:24.551511 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:24.552175 (+ 664us) server_negotiation.cc:407] Connection header received
0927 12:45:24.553451 (+ 1276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:24.553453 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:24.553503 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:24.553641 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:24.555171 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:24.556013 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:24.557127 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:24.557374 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:24.557490 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:45:24.560228 (+ 2738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:45:24.560234 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:45:24.560238 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:45:24.560272 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:45:24.561989 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:45:24.562477 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:45:24.562480 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:45:24.562481 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:45:24.562553 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:45:24.562980 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:45:24.562982 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:45:24.562984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:45:24.563164 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:45:24.563242 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 12:45:24.563861 (+ 619us) server_negotiation.cc:299] Negotiation successful
0927 12:45:24.563967 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260927 12:45:24.564986 23168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:24.551552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33161 (local address 127.0.0.1:58162)
0927 12:45:24.551998 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0927 12:45:24.552041 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 12:45:24.553218 (+ 1177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:45:24.553822 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:45:24.553833 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:45:24.554394 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:45:24.555017 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:45:24.555030 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:45:24.556122 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:45:24.556135 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:45:24.556985 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:45:24.556994 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:45:24.557528 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:45:24.557532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:45:24.557737 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:45:24.558362 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:45:24.558378 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:45:24.560023 (+ 1645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:45:24.562118 (+ 2095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:45:24.562122 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:45:24.562138 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:45:24.562389 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:45:24.562649 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:45:24.562655 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:45:24.562656 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:45:24.562896 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:45:24.563255 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:45:24.563260 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:45:24.563592 (+ 332us) client_negotiation.cc:769] Sending connection context
0927 12:45:24.563806 (+ 214us) client_negotiation.cc:240] Negotiation successful
0927 12:45:24.564067 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":112,"threads_started":1}
I20260927 12:45:24.566584 23166 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43945 } has no permanent_uuid. Determining permanent_uuid...
I20260927 12:45:24.566881 23091 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43945 } attempt: 3
Sep 27 12:45:24 dist-test-slave-bnc1 krb5kdc[23010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513124, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 27 12:45:24 dist-test-slave-bnc1 krb5kdc[23010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513124, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 12:45:24.577572 23168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:24.567160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43945 (local address 127.0.0.1:33326)
0927 12:45:24.567256 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0927 12:45:24.567271 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 12:45:24.567349 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:45:24.568251 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:45:24.568253 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:45:24.568427 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:45:24.568503 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:45:24.568508 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:45:24.569645 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:45:24.569648 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:45:24.570554 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:45:24.570565 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:45:24.571472 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:45:24.571478 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:45:24.571609 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:45:24.572544 (+ 935us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:45:24.572562 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:45:24.573901 (+ 1339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:45:24.575906 (+ 2005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:45:24.575912 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:45:24.575914 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:45:24.576158 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:45:24.576432 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:45:24.576435 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:45:24.576437 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:45:24.576531 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:45:24.577055 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:45:24.577058 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:45:24.577138 (+ 80us) client_negotiation.cc:769] Sending connection context
0927 12:45:24.577264 (+ 126us) client_negotiation.cc:240] Negotiation successful
0927 12:45:24.577395 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260927 12:45:24.577760 23171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:24.567252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33326 (local address 127.0.98.61:43945)
0927 12:45:24.567721 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:24.567733 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:24.567757 (+ 24us) server_negotiation.cc:407] Connection header received
0927 12:45:24.567800 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:24.567806 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:24.568013 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:24.568154 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:24.568611 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:24.569502 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:24.570687 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:24.571339 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:24.572267 (+ 928us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:45:24.574058 (+ 1791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:45:24.574062 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:45:24.574064 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:45:24.574091 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:45:24.575803 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:45:24.576269 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:45:24.576274 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:45:24.576275 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:45:24.576342 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:45:24.576616 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:45:24.576619 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:45:24.576620 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:45:24.576947 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:45:24.577091 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 12:45:24.577386 (+ 295us) server_negotiation.cc:299] Negotiation successful
0927 12:45:24.577524 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":191,"server-negotiator.queue_time_us":365,"spinlock_wait_cycles":9472,"thread_start_us":130,"threads_started":1}
I20260927 12:45:24.577783 23093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:24.567291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43945 (local address 127.0.0.1:33330)
0927 12:45:24.567426 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 12:45:24.567439 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:45:24.567502 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:45:24.568167 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:45:24.568173 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:45:24.568373 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:45:24.568510 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:45:24.568520 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:45:24.569779 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:45:24.569783 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:45:24.570931 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:45:24.570940 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:45:24.571448 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:45:24.571451 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:45:24.571565 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:45:24.572078 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:45:24.572090 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:45:24.573452 (+ 1362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:45:24.576151 (+ 2699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:45:24.576155 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:45:24.576156 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:45:24.576445 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:45:24.576675 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:45:24.576679 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:45:24.576681 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:45:24.576796 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:45:24.577171 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:45:24.577174 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:45:24.577274 (+ 100us) client_negotiation.cc:769] Sending connection context
0927 12:45:24.577388 (+ 114us) client_negotiation.cc:240] Negotiation successful
0927 12:45:24.577525 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 12:45:24.578207 23172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:24.567568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33330 (local address 127.0.98.61:43945)
0927 12:45:24.567821 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:24.567829 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:24.567838 (+ 9us) server_negotiation.cc:407] Connection header received
0927 12:45:24.567861 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:24.567863 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:24.568013 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:24.568153 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:24.568673 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:24.569646 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:24.571044 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:24.571333 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:24.571447 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:45:24.573682 (+ 2235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:45:24.573691 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:45:24.573702 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:45:24.573722 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:45:24.576042 (+ 2320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:45:24.576528 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:45:24.576530 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:45:24.576531 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:45:24.576582 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:45:24.576900 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:45:24.576911 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:45:24.576913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:45:24.577065 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:45:24.577173 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 12:45:24.577438 (+ 265us) server_negotiation.cc:299] Negotiation successful
0927 12:45:24.577607 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":82,"server-negotiator.queue_time_us":203,"spinlock_wait_cycles":4480,"thread_start_us":107,"threads_started":1}
I20260927 12:45:24.579584 23091 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36441 } has no permanent_uuid. Determining permanent_uuid...
I20260927 12:45:24.579751 23166 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36441 } has no permanent_uuid. Determining permanent_uuid...
I20260927 12:45:24.580322 23093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:24.580069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36441 (local address 127.0.0.1:57786)
0927 12:45:24.580186 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0927 12:45:24.580253 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260927 12:45:24.580448 23168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:24.580215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36441 (local address 127.0.0.1:57798)
0927 12:45:24.580300 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0927 12:45:24.580363 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260927 12:45:24.580655 23114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36441: connect: Connection refused (error 111)
I20260927 12:45:24.581055 23093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:24.580796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36441 (local address 127.0.0.1:57802)
0927 12:45:24.580926 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 12:45:24.580994 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260927 12:45:24.581297 23091 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36441: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36441: connect: Connection refused (error 111)
I20260927 12:45:24.581537 23168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:24.581314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36441 (local address 127.0.0.1:57812)
0927 12:45:24.581429 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0927 12:45:24.581476 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 12:45:24.581701 23166 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36441: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36441: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512840836255-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512840836255-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260927 12:45:24.582581 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
/tmp/dist-test-task_Q2zcw/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512840836255-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512840836255-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512840836255-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512840836255-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:36441
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512840836255-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33161,127.0.98.61:43945,127.0.98.60:36441
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512840836255-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512840836255-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512840836255-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 12:45:24.612480 23091 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36441 } attempt: 1
I20260927 12:45:24.613363 23093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:24.613020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36441 (local address 127.0.0.1:57824)
0927 12:45:24.613187 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 12:45:24.613262 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 12:45:24.614198 23093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:24.613920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36441 (local address 127.0.0.1:57840)
0927 12:45:24.614060 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 12:45:24.614126 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 12:45:24.614367 23091 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36441: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36441: connect: Connection refused (error 111)
I20260927 12:45:24.630965 23166 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36441 } attempt: 1
I20260927 12:45:24.631819 23168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:24.631488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36441 (local address 127.0.0.1:57852)
0927 12:45:24.631656 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0927 12:45:24.631724 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260927 12:45:24.632558 23168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:24.632344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36441 (local address 127.0.0.1:57868)
0927 12:45:24.632444 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0927 12:45:24.632494 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 12:45:24.632706 23166 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36441: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36441: connect: Connection refused (error 111)
I20260927 12:45:24.689553 23091 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36441 } attempt: 2
I20260927 12:45:24.690474 23093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:24.690122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36441 (local address 127.0.0.1:57882)
0927 12:45:24.690303 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0927 12:45:24.690377 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260927 12:45:24.691301 23093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:24.691029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36441 (local address 127.0.0.1:57898)
0927 12:45:24.691156 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 12:45:24.691220 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 12:45:24.691473 23091 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36441: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36441: connect: Connection refused (error 111)
W20260927 12:45:24.696198 23174 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 12:45:24.696466 23174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 12:45:24.696491 23174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 12:45:24.696532 23174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 12:45:24.700712 23174 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 12:45:24.700774 23174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 12:45:24.700793 23174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 12:45:24.700858 23174 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 12:45:24.700884 23174 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 12:45:24.700884 23166 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36441 } attempt: 2
I20260927 12:45:24.701756 23168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:24.701400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36441 (local address 127.0.0.1:57902)
0927 12:45:24.701584 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 12:45:24.701664 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260927 12:45:24.702478 23168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:24.702255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36441 (local address 127.0.0.1:57910)
0927 12:45:24.702363 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0927 12:45:24.702413 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 12:45:24.702639 23166 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36441: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36441: connect: Connection refused (error 111)
I20260927 12:45:24.706815 23174 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512840836255-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512840836255-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33161,127.0.98.61:43945,127.0.98.60:36441
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512840836255-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:36441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512840836255-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23174
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512840836255-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 27 Sep 2026 12:00:20 UTC on ccycloud.quasar-yylkxm.root.comops.site
build id 4676078
I20260927 12:45:24.707897 23174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 12:45:24.708906 23174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 12:45:24.717695 23182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:45:24.717716 23179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 12:45:24.717716 23180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 12:45:24.717941 23174 server_base.cc:1034] running on GCE node
I20260927 12:45:24.718660 23174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 12:45:24.719856 23174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 12:45:24.721015 23174 hybrid_clock.cc:648] HybridClock initialized: now 1790513124720975 us; error 54 us; skew 500 ppm
Sep 27 12:45:24 dist-test-slave-bnc1 krb5kdc[23010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513124, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 12:45:24.723920 23174 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260927 12:45:24.724913 23174 webserver.cc:460] Webserver started at http://127.0.98.60:41285/ using document root <none> and password file <none>
I20260927 12:45:24.725505 23174 fs_manager.cc:359] Metadata directory not provided
I20260927 12:45:24.725641 23174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 12:45:24.725876 23174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 12:45:24.728004 23174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512840836255-392-0/minicluster-data/master-2/data/instance:
uuid: "adf5c7b4dc6144fcbf399dea77a6e845"
format_stamp: "Formatted at 2026-09-27 12:45:24 on dist-test-slave-bnc1"
server_key: "8630dd6df74bcdebc53bbed05dbfe81a"
server_key_iv: "c086bae3277d89082e9adf9512d46d7f"
server_key_version: "encryptionkey@0"
I20260927 12:45:24.728616 23174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512840836255-392-0/minicluster-data/master-2/wal/instance:
uuid: "adf5c7b4dc6144fcbf399dea77a6e845"
format_stamp: "Formatted at 2026-09-27 12:45:24 on dist-test-slave-bnc1"
server_key: "8630dd6df74bcdebc53bbed05dbfe81a"
server_key_iv: "c086bae3277d89082e9adf9512d46d7f"
server_key_version: "encryptionkey@0"
I20260927 12:45:24.732440 23174 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 12:45:24.734869 23189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 12:45:24.736052 23174 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 12:45:24.736197 23174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512840836255-392-0/minicluster-data/master-2/data,/tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512840836255-392-0/minicluster-data/master-2/wal
uuid: "adf5c7b4dc6144fcbf399dea77a6e845"
format_stamp: "Formatted at 2026-09-27 12:45:24 on dist-test-slave-bnc1"
server_key: "8630dd6df74bcdebc53bbed05dbfe81a"
server_key_iv: "c086bae3277d89082e9adf9512d46d7f"
server_key_version: "encryptionkey@0"
I20260927 12:45:24.736308 23174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512840836255-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512840836255-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512840836255-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 12:45:24.749615 23174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 12:45:24.752892 23174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 12:45:24.753118 23174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 12:45:24.760419 23174 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:36441
I20260927 12:45:24.760710 23241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:36441 every 8 connection(s)
I20260927 12:45:24.761498 23174 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512840836255-392-0/minicluster-data/master-2/data/info.pb
I20260927 12:45:24.764353 23242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 12:45:24.768096 23242 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33161 } has no permanent_uuid. Determining permanent_uuid...
I20260927 12:45:24.768496 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu as pid 23174
I20260927 12:45:24.768625 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_Q2zcw/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790512840836255-392-0/minicluster-data/master-2/wal/instance
I20260927 12:45:24.768932 392 external_mini_cluster.cc:1428] Setting key ac1af747dd61e7c1ef1194fa7795c230
Sep 27 12:45:24 dist-test-slave-bnc1 krb5kdc[23010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513124, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 27 12:45:24 dist-test-slave-bnc1 krb5kdc[23010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 12:45:24.782395 23094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:24.770198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58170 (local address 127.0.98.62:33161)
0927 12:45:24.770327 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:24.770332 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:24.770911 (+ 579us) server_negotiation.cc:407] Connection header received
0927 12:45:24.771666 (+ 755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:24.771670 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:24.771716 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:24.771816 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:24.772900 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:24.773685 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:24.775014 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:24.775342 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:24.775449 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:45:24.778300 (+ 2851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:45:24.778304 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:45:24.778307 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:45:24.778340 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:45:24.779904 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:45:24.780476 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:45:24.780479 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:45:24.780480 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:45:24.780555 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:45:24.780931 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:45:24.780935 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:45:24.780936 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:45:24.781197 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:45:24.781303 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 12:45:24.782055 (+ 752us) server_negotiation.cc:299] Negotiation successful
0927 12:45:24.782206 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 12:45:24.783557 23245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:24.770305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57914 (local address 127.0.98.60:36441)
0927 12:45:24.770690 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:24.770715 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:24.770747 (+ 32us) server_negotiation.cc:407] Connection header received
0927 12:45:24.771438 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:24.771455 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:24.771697 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:24.771865 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:24.772734 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:24.773987 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:24.775103 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:24.775636 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:24.775715 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:45:24.778745 (+ 3030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:45:24.778756 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:45:24.778772 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:45:24.778802 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:45:24.780733 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:45:24.781226 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:45:24.781231 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:45:24.781232 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:45:24.781366 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:45:24.781706 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:45:24.781711 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:45:24.781712 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:45:24.782010 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:45:24.782209 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0927 12:45:24.782455 (+ 246us) server_negotiation.cc:299] Negotiation successful
0927 12:45:24.782600 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"spinlock_wait_cycles":384,"thread_start_us":136,"threads_started":1}
I20260927 12:45:24.783557 23246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:24.770305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33161 (local address 127.0.0.1:58170)
0927 12:45:24.770760 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0927 12:45:24.770805 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 12:45:24.771454 (+ 649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:45:24.771884 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:45:24.771889 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:45:24.772364 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:45:24.772759 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:45:24.772773 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:45:24.773784 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:45:24.773786 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:45:24.774895 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:45:24.774906 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:45:24.775487 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:45:24.775490 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:45:24.775642 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:45:24.776268 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:45:24.776295 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:45:24.778114 (+ 1819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:45:24.780012 (+ 1898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:45:24.780019 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:45:24.780031 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:45:24.780378 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:45:24.780655 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:45:24.780660 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:45:24.780663 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:45:24.780837 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:45:24.781322 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:45:24.781331 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:45:24.781742 (+ 411us) client_negotiation.cc:769] Sending connection context
0927 12:45:24.782005 (+ 263us) client_negotiation.cc:240] Negotiation successful
0927 12:45:24.782398 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"spinlock_wait_cycles":49664,"thread_start_us":152,"threads_started":1}
I20260927 12:45:24.784863 23242 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43945 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 12:45:24 dist-test-slave-bnc1 krb5kdc[23010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513124, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 12:45:24.790630 23091 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36441 } attempt: 3
Sep 27 12:45:24 dist-test-slave-bnc1 krb5kdc[23010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260927 12:45:24.793807 23166 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36441 } attempt: 3
I20260927 12:45:24.794400 23172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:24.785521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33344 (local address 127.0.98.61:43945)
0927 12:45:24.785692 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:24.785697 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:24.785713 (+ 16us) server_negotiation.cc:407] Connection header received
0927 12:45:24.785838 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:24.785843 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:24.785895 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:24.785980 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:24.786408 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:24.786993 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:24.787839 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:24.788129 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:24.788252 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:45:24.790410 (+ 2158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:45:24.790414 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:45:24.790417 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:45:24.790448 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:45:24.792345 (+ 1897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:45:24.792862 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:45:24.792870 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:45:24.792873 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:45:24.792967 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:45:24.793317 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:45:24.793320 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:45:24.793322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:45:24.793596 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:45:24.793696 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 12:45:24.794112 (+ 416us) server_negotiation.cc:299] Negotiation successful
0927 12:45:24.794234 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 12:45:24.794821 23246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:24.785409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43945 (local address 127.0.0.1:33344)
0927 12:45:24.785612 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0927 12:45:24.785626 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 12:45:24.785704 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:45:24.786006 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:45:24.786009 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:45:24.786179 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:45:24.786282 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:45:24.786288 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:45:24.787109 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:45:24.787116 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:45:24.787702 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:45:24.787710 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:45:24.788247 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:45:24.788251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:45:24.788356 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:45:24.788891 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:45:24.788910 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:45:24.790275 (+ 1365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:45:24.792481 (+ 2206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:45:24.792487 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:45:24.792491 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:45:24.792749 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:45:24.793086 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:45:24.793089 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:45:24.793091 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:45:24.793186 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:45:24.793798 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:45:24.793802 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:45:24.793903 (+ 101us) client_negotiation.cc:769] Sending connection context
0927 12:45:24.794592 (+ 689us) client_negotiation.cc:240] Negotiation successful
0927 12:45:24.794710 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109}
I20260927 12:45:24.795383 23094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:24.786000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58186 (local address 127.0.98.62:33161)
0927 12:45:24.786125 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:24.786128 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:24.786219 (+ 91us) server_negotiation.cc:407] Connection header received
0927 12:45:24.786361 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:24.786364 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:24.786401 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:24.786488 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:24.787046 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:24.787520 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:24.788293 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:24.788539 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:24.788648 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:45:24.791383 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:45:24.791388 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:45:24.791391 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:45:24.791418 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:45:24.793161 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:45:24.793670 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:45:24.793673 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:45:24.793675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:45:24.793768 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:45:24.794042 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:45:24.794044 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:45:24.794046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:45:24.794267 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:45:24.794470 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0927 12:45:24.795093 (+ 623us) server_negotiation.cc:299] Negotiation successful
0927 12:45:24.795222 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 12:45:24.795490 23242 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36441 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 12:45:24 dist-test-slave-bnc1 krb5kdc[23010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513124, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260927 12:45:24.801183 23093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:24.791038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36441 (local address 127.0.0.1:57916)
0927 12:45:24.791131 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0927 12:45:24.791153 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 12:45:24.791219 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:45:24.791908 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:45:24.791910 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:45:24.792096 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:45:24.792174 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:45:24.792178 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:45:24.793319 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:45:24.793322 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:45:24.794057 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:45:24.794065 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:45:24.795189 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:45:24.795194 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:45:24.795271 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:45:24.795792 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:45:24.795807 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:45:24.797280 (+ 1473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:45:24.799310 (+ 2030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:45:24.799313 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:45:24.799315 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:45:24.799561 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:45:24.799839 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:45:24.799842 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:45:24.799844 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:45:24.799923 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:45:24.800629 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:45:24.800632 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:45:24.800744 (+ 112us) client_negotiation.cc:769] Sending connection context
0927 12:45:24.800873 (+ 129us) client_negotiation.cc:240] Negotiation successful
0927 12:45:24.801019 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260927 12:45:24.801502 23245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:24.791357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57916 (local address 127.0.98.60:36441)
0927 12:45:24.791515 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:24.791519 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:24.791531 (+ 12us) server_negotiation.cc:407] Connection header received
0927 12:45:24.791565 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:24.791567 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:24.791598 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:24.791694 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:24.792506 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:24.793200 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:24.794228 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:24.795073 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:24.795651 (+ 578us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:45:24.797423 (+ 1772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:45:24.797427 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:45:24.797429 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:45:24.797452 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:45:24.799191 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:45:24.799669 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:45:24.799672 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:45:24.799674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:45:24.799746 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:45:24.800030 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:45:24.800035 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:45:24.800040 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:45:24.800257 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:45:24.800619 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0927 12:45:24.801219 (+ 600us) server_negotiation.cc:299] Negotiation successful
0927 12:45:24.801341 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
Sep 27 12:45:24 dist-test-slave-bnc1 krb5kdc[23010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513124, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 12:45:24 dist-test-slave-bnc1 krb5kdc[23010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513124, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 27 12:45:24 dist-test-slave-bnc1 krb5kdc[23010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790513124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260927 12:45:24.805305 23091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3add08c79334ab2895224604e7531e3: Bootstrap starting.
I20260927 12:45:24.806197 23246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:24.796083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36441 (local address 127.0.0.1:57924)
0927 12:45:24.796301 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0927 12:45:24.796314 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:45:24.796385 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:45:24.797022 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:45:24.797025 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:45:24.797194 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:45:24.797284 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:45:24.797291 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:45:24.798579 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:45:24.798586 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:45:24.799530 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:45:24.799539 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:45:24.800111 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:45:24.800115 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:45:24.800202 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:45:24.800598 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:45:24.800611 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:45:24.802430 (+ 1819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:45:24.804611 (+ 2181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:45:24.804613 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:45:24.804616 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:45:24.804849 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:45:24.805162 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:45:24.805166 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:45:24.805169 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:45:24.805253 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:45:24.805689 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:45:24.805695 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:45:24.805788 (+ 93us) client_negotiation.cc:769] Sending connection context
0927 12:45:24.805870 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 12:45:24.806016 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":48}
I20260927 12:45:24.806857 23248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:24.796463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57924 (local address 127.0.98.60:36441)
0927 12:45:24.796763 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:24.796773 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:24.796785 (+ 12us) server_negotiation.cc:407] Connection header received
0927 12:45:24.796820 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:24.796822 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:24.796865 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:24.797391 (+ 526us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:24.797533 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:24.798447 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:24.799662 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:24.800012 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:24.800380 (+ 368us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:45:24.802547 (+ 2167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:45:24.802550 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:45:24.802552 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:45:24.802571 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:45:24.804495 (+ 1924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:45:24.804972 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:45:24.804976 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:45:24.804980 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:45:24.805067 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:45:24.805364 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:45:24.805366 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:45:24.805368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:45:24.805564 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:45:24.806457 (+ 893us) server_negotiation.cc:1035] Waiting for connection context
0927 12:45:24.806538 (+ 81us) server_negotiation.cc:299] Negotiation successful
0927 12:45:24.806678 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":134,"threads_started":1}
I20260927 12:45:24.807893 23168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:24.794166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36441 (local address 127.0.0.1:57922)
0927 12:45:24.794325 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 12:45:24.794338 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 12:45:24.794404 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 12:45:24.796204 (+ 1800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 12:45:24.796210 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 12:45:24.796393 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0927 12:45:24.796491 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:45:24.796498 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:45:24.798226 (+ 1728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:45:24.798233 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:45:24.798969 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 12:45:24.798986 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 12:45:24.799849 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 12:45:24.799854 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 12:45:24.800724 (+ 870us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:45:24.802032 (+ 1308us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 12:45:24.802045 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 12:45:24.803563 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 12:45:24.806213 (+ 2650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:45:24.806216 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:45:24.806219 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:45:24.806464 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:45:24.806806 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 12:45:24.806809 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 12:45:24.806811 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 12:45:24.806903 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 12:45:24.807386 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 12:45:24.807390 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 12:45:24.807514 (+ 124us) client_negotiation.cc:769] Sending connection context
0927 12:45:24.807597 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 12:45:24.807732 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"mutex_wait_us":792}
I20260927 12:45:24.808672 23247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:24.794749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57922 (local address 127.0.98.60:36441)
0927 12:45:24.795954 (+ 1205us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:24.795964 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:24.795976 (+ 12us) server_negotiation.cc:407] Connection header received
0927 12:45:24.796008 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:24.796011 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:24.796047 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:24.796607 (+ 560us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:24.796721 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:24.798073 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:24.799103 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:24.799703 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:24.800008 (+ 305us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:45:24.803729 (+ 3721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:45:24.803733 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:45:24.803736 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:45:24.803761 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:45:24.806044 (+ 2283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:45:24.806592 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:45:24.806596 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:45:24.806598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:45:24.806686 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:45:24.807045 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:45:24.807050 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:45:24.807054 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:45:24.807278 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:45:24.808337 (+ 1059us) server_negotiation.cc:1035] Waiting for connection context
0927 12:45:24.808412 (+ 75us) server_negotiation.cc:299] Negotiation successful
0927 12:45:24.808522 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":65,"server-negotiator.queue_time_us":1134,"spinlock_wait_cycles":768,"thread_start_us":129,"threads_started":1}
I20260927 12:45:24.809170 23172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:24.797301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33358 (local address 127.0.98.61:43945)
0927 12:45:24.797416 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:24.797422 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:24.797623 (+ 201us) server_negotiation.cc:407] Connection header received
0927 12:45:24.798127 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:24.798130 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:24.798172 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:24.798224 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:24.799951 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:24.800633 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:24.801993 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:24.802317 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:24.802419 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:45:24.805533 (+ 3114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:45:24.805537 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:45:24.805540 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:45:24.805561 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:45:24.807200 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:45:24.807666 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:45:24.807669 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:45:24.807671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:45:24.807755 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:45:24.808077 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:45:24.808082 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:45:24.808083 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:45:24.808286 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:45:24.808423 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0927 12:45:24.808935 (+ 512us) server_negotiation.cc:299] Negotiation successful
0927 12:45:24.809043 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":49}
I20260927 12:45:24.810422 23242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adf5c7b4dc6144fcbf399dea77a6e845: Bootstrap starting.
I20260927 12:45:24.811623 23166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f326d8addfb42388cc16d0f9559eb93: Bootstrap starting.
I20260927 12:45:24.812459 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 12:45:24.813235 23242 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P adf5c7b4dc6144fcbf399dea77a6e845: Neither blocks nor log segments found. Creating new log.
I20260927 12:45:24.814081 23166 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f326d8addfb42388cc16d0f9559eb93: Neither blocks nor log segments found. Creating new log.
I20260927 12:45:24.814182 23242 log.cc:824] T 00000000000000000000000000000000 P adf5c7b4dc6144fcbf399dea77a6e845: Log is configured to *not* fsync() on all Append() calls
I20260927 12:45:24.813539 23065 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "c3add08c79334ab2895224604e7531e3"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58186
I20260927 12:45:24.814911 23166 log.cc:824] T 00000000000000000000000000000000 P 7f326d8addfb42388cc16d0f9559eb93: Log is configured to *not* fsync() on all Append() calls
I20260927 12:45:24.816159 23091 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3add08c79334ab2895224604e7531e3: Neither blocks nor log segments found. Creating new log.
I20260927 12:45:24.816943 23091 log.cc:824] T 00000000000000000000000000000000 P c3add08c79334ab2895224604e7531e3: Log is configured to *not* fsync() on all Append() calls
I20260927 12:45:24.817113 23166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f326d8addfb42388cc16d0f9559eb93: No bootstrap required, opened a new log
I20260927 12:45:24.818598 23242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adf5c7b4dc6144fcbf399dea77a6e845: No bootstrap required, opened a new log
I20260927 12:45:24.821239 23166 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f326d8addfb42388cc16d0f9559eb93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3add08c79334ab2895224604e7531e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33161 } } peers { permanent_uuid: "7f326d8addfb42388cc16d0f9559eb93" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43945 } } peers { permanent_uuid: "adf5c7b4dc6144fcbf399dea77a6e845" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36441 } }
I20260927 12:45:24.821470 23166 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f326d8addfb42388cc16d0f9559eb93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f326d8addfb42388cc16d0f9559eb93, State: Initialized, Role: FOLLOWER
I20260927 12:45:24.821832 23166 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f326d8addfb42388cc16d0f9559eb93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3add08c79334ab2895224604e7531e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33161 } } peers { permanent_uuid: "7f326d8addfb42388cc16d0f9559eb93" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43945 } } peers { permanent_uuid: "adf5c7b4dc6144fcbf399dea77a6e845" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36441 } }
I20260927 12:45:24.821943 23091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3add08c79334ab2895224604e7531e3: No bootstrap required, opened a new log
I20260927 12:45:24.822352 23252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f326d8addfb42388cc16d0f9559eb93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3add08c79334ab2895224604e7531e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33161 } } peers { permanent_uuid: "7f326d8addfb42388cc16d0f9559eb93" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43945 } } peers { permanent_uuid: "adf5c7b4dc6144fcbf399dea77a6e845" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36441 } } }
I20260927 12:45:24.822489 23252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f326d8addfb42388cc16d0f9559eb93 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 12:45:24.823220 23166 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f326d8addfb42388cc16d0f9559eb93 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:45:24.824992 23091 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3add08c79334ab2895224604e7531e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3add08c79334ab2895224604e7531e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33161 } } peers { permanent_uuid: "7f326d8addfb42388cc16d0f9559eb93" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43945 } } peers { permanent_uuid: "adf5c7b4dc6144fcbf399dea77a6e845" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36441 } }
I20260927 12:45:24.825250 23091 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3add08c79334ab2895224604e7531e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3add08c79334ab2895224604e7531e3, State: Initialized, Role: FOLLOWER
I20260927 12:45:24.825623 23091 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3add08c79334ab2895224604e7531e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3add08c79334ab2895224604e7531e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33161 } } peers { permanent_uuid: "7f326d8addfb42388cc16d0f9559eb93" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43945 } } peers { permanent_uuid: "adf5c7b4dc6144fcbf399dea77a6e845" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36441 } }
I20260927 12:45:24.826902 23091 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3add08c79334ab2895224604e7531e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:45:24.826984 23065 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3add08c79334ab2895224604e7531e3 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 12:45:24.827098 23065 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3add08c79334ab2895224604e7531e3 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:45:24.828502 23065 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3add08c79334ab2895224604e7531e3 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3add08c79334ab2895224604e7531e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33161 } } peers { permanent_uuid: "7f326d8addfb42388cc16d0f9559eb93" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43945 } } peers { permanent_uuid: "adf5c7b4dc6144fcbf399dea77a6e845" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36441 } }
I20260927 12:45:24.828526 23242 raft_consensus.cc:348] T 00000000000000000000000000000000 P adf5c7b4dc6144fcbf399dea77a6e845 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3add08c79334ab2895224604e7531e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33161 } } peers { permanent_uuid: "7f326d8addfb42388cc16d0f9559eb93" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43945 } } peers { permanent_uuid: "adf5c7b4dc6144fcbf399dea77a6e845" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36441 } }
I20260927 12:45:24.828742 23242 raft_consensus.cc:729] T 00000000000000000000000000000000 P adf5c7b4dc6144fcbf399dea77a6e845 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adf5c7b4dc6144fcbf399dea77a6e845, State: Initialized, Role: FOLLOWER
I20260927 12:45:24.829036 23242 consensus_queue.cc:260] T 00000000000000000000000000000000 P adf5c7b4dc6144fcbf399dea77a6e845 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3add08c79334ab2895224604e7531e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33161 } } peers { permanent_uuid: "7f326d8addfb42388cc16d0f9559eb93" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43945 } } peers { permanent_uuid: "adf5c7b4dc6144fcbf399dea77a6e845" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36441 } }
I20260927 12:45:24.829434 23065 leader_election.cc:288] T 00000000000000000000000000000000 P c3add08c79334ab2895224604e7531e3 [CANDIDATE]: Term 1 election: Requested vote from peers 7f326d8addfb42388cc16d0f9559eb93 (127.0.98.61:43945), adf5c7b4dc6144fcbf399dea77a6e845 (127.0.98.60:36441)
I20260927 12:45:24.829524 23275 sys_catalog.cc:455] T 00000000000000000000000000000000 P adf5c7b4dc6144fcbf399dea77a6e845 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3add08c79334ab2895224604e7531e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33161 } } peers { permanent_uuid: "7f326d8addfb42388cc16d0f9559eb93" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43945 } } peers { permanent_uuid: "adf5c7b4dc6144fcbf399dea77a6e845" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36441 } } }
I20260927 12:45:24.829708 23275 sys_catalog.cc:458] T 00000000000000000000000000000000 P adf5c7b4dc6144fcbf399dea77a6e845 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 12:45:24.830346 23242 sys_catalog.cc:564] T 00000000000000000000000000000000 P adf5c7b4dc6144fcbf399dea77a6e845 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 12:45:24.830511 23139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c3add08c79334ab2895224604e7531e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7f326d8addfb42388cc16d0f9559eb93"
I20260927 12:45:24.830638 23216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c3add08c79334ab2895224604e7531e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "adf5c7b4dc6144fcbf399dea77a6e845"
I20260927 12:45:24.830735 23139 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f326d8addfb42388cc16d0f9559eb93 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:45:24.830821 23216 raft_consensus.cc:3037] T 00000000000000000000000000000000 P adf5c7b4dc6144fcbf399dea77a6e845 [term 0 FOLLOWER]: Advancing to term 1
I20260927 12:45:24.832049 23216 raft_consensus.cc:2454] T 00000000000000000000000000000000 P adf5c7b4dc6144fcbf399dea77a6e845 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3add08c79334ab2895224604e7531e3 in term 1.
I20260927 12:45:24.832049 23139 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7f326d8addfb42388cc16d0f9559eb93 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3add08c79334ab2895224604e7531e3 in term 1.
I20260927 12:45:24.832475 23039 leader_election.cc:302] T 00000000000000000000000000000000 P c3add08c79334ab2895224604e7531e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: adf5c7b4dc6144fcbf399dea77a6e845, c3add08c79334ab2895224604e7531e3; no voters:
W20260927 12:45:24.833920 23263 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7f326d8addfb42388cc16d0f9559eb93: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 12:45:24.834053 23263 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 12:45:24.827349 23098 master_runner.cc:186] Error getting master registration for 127.0.98.62:33161: OK, instance_id { permanent_uuid: "c3add08c79334ab2895224604e7531e3" instance_seqno: 1790513124325242 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 12:45:24.836835 23098 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260927 12:45:24.836851 23278 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c3add08c79334ab2895224604e7531e3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260927 12:45:24.837066 23293 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3add08c79334ab2895224604e7531e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3add08c79334ab2895224604e7531e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33161 } } peers { permanent_uuid: "7f326d8addfb42388cc16d0f9559eb93" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43945 } } peers { permanent_uuid: "adf5c7b4dc6144fcbf399dea77a6e845" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36441 } } }
I20260927 12:45:24.837179 23293 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3add08c79334ab2895224604e7531e3 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 12:45:24.837251 23293 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c3add08c79334ab2895224604e7531e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 12:45:24.837323 23293 raft_consensus.cc:686] T 00000000000000000000000000000000 P c3add08c79334ab2895224604e7531e3 [term 1 LEADER]: Becoming Leader. State: Replica: c3add08c79334ab2895224604e7531e3, State: Running, Role: LEADER
W20260927 12:45:24.836913 23278 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 12:45:24.840118 23023 master_runner.cc:263] No leader master found from master c3add08c79334ab2895224604e7531e3
I20260927 12:45:24.840206 23023 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 12:45:24.842108 23174 master_runner.cc:263] No leader master found from master adf5c7b4dc6144fcbf399dea77a6e845
I20260927 12:45:24.842185 23174 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 12:45:24.841351 23293 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3add08c79334ab2895224604e7531e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3add08c79334ab2895224604e7531e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33161 } } peers { permanent_uuid: "7f326d8addfb42388cc16d0f9559eb93" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43945 } } peers { permanent_uuid: "adf5c7b4dc6144fcbf399dea77a6e845" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36441 } }
I20260927 12:45:24.844159 23293 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3add08c79334ab2895224604e7531e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3add08c79334ab2895224604e7531e3. Latest consensus state: current_term: 1 leader_uuid: "c3add08c79334ab2895224604e7531e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3add08c79334ab2895224604e7531e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33161 } } peers { permanent_uuid: "7f326d8addfb42388cc16d0f9559eb93" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43945 } } peers { permanent_uuid: "adf5c7b4dc6144fcbf399dea77a6e845" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36441 } } }
I20260927 12:45:24.844283 23293 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3add08c79334ab2895224604e7531e3 [sys.catalog]: This master's current role is: LEADER
W20260927 12:45:24.848258 23288 catalog_manager.cc:1558] T 00000000000000000000000000000000 P adf5c7b4dc6144fcbf399dea77a6e845: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 12:45:24.848397 23288 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 12:45:24.854046 23094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:24.834679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58196 (local address 127.0.98.62:33161)
0927 12:45:24.835215 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:24.835219 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:24.837463 (+ 2244us) server_negotiation.cc:407] Connection header received
0927 12:45:24.837598 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:24.837602 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:24.837652 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:24.837724 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:24.838756 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:24.839585 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:24.841696 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:24.842004 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:24.842073 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:45:24.850061 (+ 7988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:45:24.850066 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:45:24.850069 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:45:24.850098 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:45:24.851977 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:45:24.852509 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:45:24.852514 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:45:24.852515 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:45:24.852604 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:45:24.852946 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:45:24.852951 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:45:24.852956 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:45:24.853200 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:45:24.853328 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 12:45:24.853757 (+ 429us) server_negotiation.cc:299] Negotiation successful
0927 12:45:24.853887 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460}
I20260927 12:45:24.854753 23247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:24.835319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57928 (local address 127.0.98.60:36441)
0927 12:45:24.835422 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:24.835426 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:24.837785 (+ 2359us) server_negotiation.cc:407] Connection header received
0927 12:45:24.837952 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:24.837956 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:24.838004 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:24.838056 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:24.838880 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:24.839622 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:24.841352 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:24.841706 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:24.841779 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:45:24.851118 (+ 9339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:45:24.851125 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:45:24.851143 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:45:24.851176 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:45:24.852886 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:45:24.853358 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:45:24.853364 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:45:24.853365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:45:24.853450 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:45:24.853773 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:45:24.853777 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:45:24.853778 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:45:24.853980 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:45:24.854076 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 12:45:24.854461 (+ 385us) server_negotiation.cc:299] Negotiation successful
0927 12:45:24.854582 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 12:45:24.857040 23299 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 12:45:24.858343 23299 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 12:45:24.860744 23172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:24.834958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33362 (local address 127.0.98.61:43945)
0927 12:45:24.835067 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:24.835072 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:24.844705 (+ 9633us) server_negotiation.cc:407] Connection header received
0927 12:45:24.844873 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:24.844876 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:24.844956 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:24.845062 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:24.846556 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:24.847361 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:24.848202 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:24.848753 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:24.848856 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:45:24.857095 (+ 8239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 12:45:24.857102 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 12:45:24.857106 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 12:45:24.857153 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 12:45:24.858834 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:45:24.859348 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:45:24.859351 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:45:24.859353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:45:24.859438 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 12:45:24.859731 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 12:45:24.859734 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 12:45:24.859736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 12:45:24.859965 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 12:45:24.860073 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 12:45:24.860443 (+ 370us) server_negotiation.cc:299] Negotiation successful
0927 12:45:24.860584 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 12:45:24.864187 23216 raft_consensus.cc:1261] T 00000000000000000000000000000000 P adf5c7b4dc6144fcbf399dea77a6e845 [term 1 FOLLOWER]: Refusing update from remote peer c3add08c79334ab2895224604e7531e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:45:24.864421 23139 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7f326d8addfb42388cc16d0f9559eb93 [term 1 FOLLOWER]: Refusing update from remote peer c3add08c79334ab2895224604e7531e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 12:45:24.864977 23264 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c3add08c79334ab2895224604e7531e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "adf5c7b4dc6144fcbf399dea77a6e845" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:45:24.866102 23293 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c3add08c79334ab2895224604e7531e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f326d8addfb42388cc16d0f9559eb93" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 12:45:24.870035 23252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f326d8addfb42388cc16d0f9559eb93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3add08c79334ab2895224604e7531e3. Latest consensus state: current_term: 1 leader_uuid: "c3add08c79334ab2895224604e7531e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3add08c79334ab2895224604e7531e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33161 } } peers { permanent_uuid: "7f326d8addfb42388cc16d0f9559eb93" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43945 } } peers { permanent_uuid: "adf5c7b4dc6144fcbf399dea77a6e845" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36441 } } }
I20260927 12:45:24.870163 23252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f326d8addfb42388cc16d0f9559eb93 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 12:45:24.870464 23275 sys_catalog.cc:455] T 00000000000000000000000000000000 P adf5c7b4dc6144fcbf399dea77a6e845 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3add08c79334ab2895224604e7531e3. Latest consensus state: current_term: 1 leader_uuid: "c3add08c79334ab2895224604e7531e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3add08c79334ab2895224604e7531e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33161 } } peers { permanent_uuid: "7f326d8addfb42388cc16d0f9559eb93" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43945 } } peers { permanent_uuid: "adf5c7b4dc6144fcbf399dea77a6e845" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36441 } } }
I20260927 12:45:24.870595 23275 sys_catalog.cc:458] T 00000000000000000000000000000000 P adf5c7b4dc6144fcbf399dea77a6e845 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 12:45:24.888172 23275 sys_catalog.cc:455] T 00000000000000000000000000000000 P adf5c7b4dc6144fcbf399dea77a6e845 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c3add08c79334ab2895224604e7531e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3add08c79334ab2895224604e7531e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33161 } } peers { permanent_uuid: "7f326d8addfb42388cc16d0f9559eb93" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43945 } } peers { permanent_uuid: "adf5c7b4dc6144fcbf399dea77a6e845" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36441 } } }
I20260927 12:45:24.888329 23275 sys_catalog.cc:458] T 00000000000000000000000000000000 P adf5c7b4dc6144fcbf399dea77a6e845 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 12:45:24.889060 23293 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3add08c79334ab2895224604e7531e3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c3add08c79334ab2895224604e7531e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3add08c79334ab2895224604e7531e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33161 } } peers { permanent_uuid: "7f326d8addfb42388cc16d0f9559eb93" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43945 } } peers { permanent_uuid: "adf5c7b4dc6144fcbf399dea77a6e845" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36441 } } }
I20260927 12:45:24.889199 23293 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3add08c79334ab2895224604e7531e3 [sys.catalog]: This master's current role is: LEADER
I20260927 12:45:24.889295 23293 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3add08c79334ab2895224604e7531e3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c3add08c79334ab2895224604e7531e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3add08c79334ab2895224604e7531e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33161 } } peers { permanent_uuid: "7f326d8addfb42388cc16d0f9559eb93" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43945 } } peers { permanent_uuid: "adf5c7b4dc6144fcbf399dea77a6e845" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36441 } } }
I20260927 12:45:24.889354 23293 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3add08c79334ab2895224604e7531e3 [sys.catalog]: This master's current role is: LEADER
I20260927 12:45:24.889389 23252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f326d8addfb42388cc16d0f9559eb93 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c3add08c79334ab2895224604e7531e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3add08c79334ab2895224604e7531e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33161 } } peers { permanent_uuid: "7f326d8addfb42388cc16d0f9559eb93" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43945 } } peers { permanent_uuid: "adf5c7b4dc6144fcbf399dea77a6e845" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36441 } } }
I20260927 12:45:24.889513 23252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f326d8addfb42388cc16d0f9559eb93 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 12:45:24.889632 23303 mvcc.cc:205] Tried to move back new op lower bound from 7333941759437066240 to 7333941759357403136. Current Snapshot: MvccSnapshot[applied={T|T < 7333941759437066240}]
I20260927 12:45:24.891542 23301 mvcc.cc:205] Tried to move back new op lower bound from 7333941759437066240 to 7333941759357403136. Current Snapshot: MvccSnapshot[applied={T|T < 7333941759437066240}]
I20260927 12:45:24.891798 23302 mvcc.cc:205] Tried to move back new op lower bound from 7333941759437066240 to 7333941759357403136. Current Snapshot: MvccSnapshot[applied={T|T < 7333941759437066240}]
I20260927 12:45:24.895321 23299 catalog_manager.cc:1347] Generated new cluster ID: c8a6e1082c68483bbc09ab46db75dfbc
I20260927 12:45:24.895388 23299 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 12:45:24.926872 23299 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 12:45:24.928205 23299 catalog_manager.cc:1504] Loading token signing keys...
I20260927 12:45:24.948596 23299 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c3add08c79334ab2895224604e7531e3: Generated new TSK 0
I20260927 12:45:24.949596 23299 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 12:45:24.976960 23094 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:45:24.973707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58210 (local address 127.0.98.62:33161)
0927 12:45:24.973843 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:24.973848 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:24.975783 (+ 1935us) server_negotiation.cc:407] Connection header received
0927 12:45:24.976001 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:24.976005 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:24.976060 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:24.976116 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:24.976802 (+ 686us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58210: BlockingRecv error: recv got EOF from 127.0.0.1:58210 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260927 12:45:24.978513 23172 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:45:24.974490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33372 (local address 127.0.98.61:43945)
0927 12:45:24.974608 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:24.974612 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:24.977505 (+ 2893us) server_negotiation.cc:407] Connection header received
0927 12:45:24.977738 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:24.977742 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:24.977796 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:24.977850 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:24.978369 (+ 519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33372: BlockingRecv error: recv got EOF from 127.0.0.1:33372 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260927 12:45:24.980129 23247 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:45:24.975170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57932 (local address 127.0.98.60:36441)
0927 12:45:24.975285 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:24.975290 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:24.978905 (+ 3615us) server_negotiation.cc:407] Connection header received
0927 12:45:24.979245 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:24.979249 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:24.979304 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:24.979358 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:24.980003 (+ 645us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57932: BlockingRecv error: recv got EOF from 127.0.0.1:57932 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260927 12:45:24.999342 23172 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:45:24.993150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33378 (local address 127.0.98.61:43945)
0927 12:45:24.993307 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:24.993313 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:24.998024 (+ 4711us) server_negotiation.cc:407] Connection header received
0927 12:45:24.998358 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:24.998363 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:24.998453 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:24.998518 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:24.999207 (+ 689us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33378: BlockingRecv error: recv got EOF from 127.0.0.1:33378 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260927 12:45:25.007011 23172 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:45:25.005158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33394 (local address 127.0.98.61:43945)
0927 12:45:25.005321 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:25.005326 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:25.005937 (+ 611us) server_negotiation.cc:407] Connection header received
0927 12:45:25.006176 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:25.006180 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:25.006238 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:25.006295 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:25.006869 (+ 574us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33394: BlockingRecv error: recv got EOF from 127.0.0.1:33394 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
W20260927 12:45:25.013355 23172 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:45:25.011439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33400 (local address 127.0.98.61:43945)
0927 12:45:25.011549 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:25.011554 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:25.012178 (+ 624us) server_negotiation.cc:407] Connection header received
0927 12:45:25.012452 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:25.012456 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:25.012515 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:25.012573 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:25.013213 (+ 640us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33400: BlockingRecv error: recv got EOF from 127.0.0.1:33400 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260927 12:45:25.021808 23172 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:45:25.019797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33404 (local address 127.0.98.61:43945)
0927 12:45:25.019934 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:25.019939 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:25.020554 (+ 615us) server_negotiation.cc:407] Connection header received
0927 12:45:25.020818 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:25.020822 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:25.020879 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:25.020960 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:25.021693 (+ 733us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33404: BlockingRecv error: recv got EOF from 127.0.0.1:33404 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260927 12:45:25.034063 23172 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:45:25.032302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33420 (local address 127.0.98.61:43945)
0927 12:45:25.032422 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:25.032427 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:25.032996 (+ 569us) server_negotiation.cc:407] Connection header received
0927 12:45:25.033219 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:25.033223 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:25.033281 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:25.033339 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:25.033916 (+ 577us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33420: BlockingRecv error: recv got EOF from 127.0.0.1:33420 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260927 12:45:25.054170 23172 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:45:25.052518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33424 (local address 127.0.98.61:43945)
0927 12:45:25.052637 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:25.052641 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:25.053184 (+ 543us) server_negotiation.cc:407] Connection header received
0927 12:45:25.053377 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:25.053381 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:25.053433 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:25.053485 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:25.054036 (+ 551us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33424: BlockingRecv error: recv got EOF from 127.0.0.1:33424 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260927 12:45:25.090147 23172 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:45:25.088858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33430 (local address 127.0.98.61:43945)
0927 12:45:25.089044 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:25.089049 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:25.089207 (+ 158us) server_negotiation.cc:407] Connection header received
0927 12:45:25.089384 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:25.089389 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:25.089445 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:25.089541 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:25.090025 (+ 484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33430: BlockingRecv error: recv got EOF from 127.0.0.1:33430 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260927 12:45:25.157341 23172 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:45:25.156275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33442 (local address 127.0.98.61:43945)
0927 12:45:25.156412 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:25.156415 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:25.156506 (+ 91us) server_negotiation.cc:407] Connection header received
0927 12:45:25.156628 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:25.156631 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:25.156677 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:25.156762 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:25.157239 (+ 477us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33442: BlockingRecv error: recv got EOF from 127.0.0.1:33442 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260927 12:45:25.288378 23172 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:45:25.287400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33456 (local address 127.0.98.61:43945)
0927 12:45:25.287531 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:25.287534 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:25.287654 (+ 120us) server_negotiation.cc:407] Connection header received
0927 12:45:25.287751 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:25.287754 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:25.287800 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:25.287884 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:25.288283 (+ 399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33456: BlockingRecv error: recv got EOF from 127.0.0.1:33456 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260927 12:45:25.547780 23172 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 12:45:25.546716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33472 (local address 127.0.98.61:43945)
0927 12:45:25.546872 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:25.546876 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:25.546900 (+ 24us) server_negotiation.cc:407] Connection header received
0927 12:45:25.547053 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:25.547057 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:25.547114 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:25.547204 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 12:45:25.547670 (+ 466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33472: BlockingRecv error: recv got EOF from 127.0.0.1:33472 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 12:45:25.835424 23263 catalog_manager.cc:1259] Loaded cluster ID: c8a6e1082c68483bbc09ab46db75dfbc
I20260927 12:45:25.835503 23263 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7f326d8addfb42388cc16d0f9559eb93: loading cluster ID for follower catalog manager: success
I20260927 12:45:25.837164 23263 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7f326d8addfb42388cc16d0f9559eb93: acquiring CA information for follower catalog manager: success
I20260927 12:45:25.838346 23263 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7f326d8addfb42388cc16d0f9559eb93: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 12:45:25.849915 23288 catalog_manager.cc:1259] Loaded cluster ID: c8a6e1082c68483bbc09ab46db75dfbc
I20260927 12:45:25.849997 23288 catalog_manager.cc:1552] T 00000000000000000000000000000000 P adf5c7b4dc6144fcbf399dea77a6e845: loading cluster ID for follower catalog manager: success
I20260927 12:45:25.851794 23288 catalog_manager.cc:1574] T 00000000000000000000000000000000 P adf5c7b4dc6144fcbf399dea77a6e845: acquiring CA information for follower catalog manager: success
I20260927 12:45:25.852873 23288 catalog_manager.cc:1602] T 00000000000000000000000000000000 P adf5c7b4dc6144fcbf399dea77a6e845: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 12:45:26.065907 23376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:26.061904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33482 (local address 127.0.98.61:43945)
0927 12:45:26.062185 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:26.062201 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:26.062221 (+ 20us) server_negotiation.cc:407] Connection header received
0927 12:45:26.062396 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:26.062399 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:26.062545 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:26.062676 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:45:26.062948 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:26.063468 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:26.064364 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:26.064601 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:26.064715 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:45:26.065040 (+ 325us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:45:26.065476 (+ 436us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:45:26.065647 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 12:45:26.065689 (+ 42us) server_negotiation.cc:299] Negotiation successful
0927 12:45:26.065745 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":86,"threads_started":1}
I20260927 12:45:26.072592 23382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:26.068829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57936 (local address 127.0.98.60:36441)
0927 12:45:26.069073 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:26.069077 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:26.069092 (+ 15us) server_negotiation.cc:407] Connection header received
0927 12:45:26.069266 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:26.069269 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:26.069393 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:26.069520 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:45:26.069755 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:26.070267 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:26.071086 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:26.071344 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:26.071452 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:45:26.071758 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:45:26.072184 (+ 426us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:45:26.072328 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 12:45:26.072365 (+ 37us) server_negotiation.cc:299] Negotiation successful
0927 12:45:26.072420 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":63,"threads_started":1}
I20260927 12:45:26.080142 23382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:26.075867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57938 (local address 127.0.98.60:36441)
0927 12:45:26.076272 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:26.076277 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:26.076289 (+ 12us) server_negotiation.cc:407] Connection header received
0927 12:45:26.076536 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:26.076539 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:26.076599 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:26.076667 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:45:26.077035 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:26.077858 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:26.078945 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:26.079251 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:26.079349 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:45:26.079561 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:45:26.079671 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:45:26.079742 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 12:45:26.079901 (+ 159us) server_negotiation.cc:299] Negotiation successful
0927 12:45:26.079956 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320}
I20260927 12:45:26.080221 23376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:26.076277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33494 (local address 127.0.98.61:43945)
0927 12:45:26.076397 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:26.076401 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:26.076536 (+ 135us) server_negotiation.cc:407] Connection header received
0927 12:45:26.076605 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:26.076608 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:26.076662 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:26.076728 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:45:26.077082 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:26.077859 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:26.078814 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:26.079165 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:26.079273 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:45:26.079692 (+ 419us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:45:26.079772 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:45:26.079859 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 12:45:26.080001 (+ 142us) server_negotiation.cc:299] Negotiation successful
0927 12:45:26.080059 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260927 12:45:26.080377 23388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 12:45:26.075773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58220 (local address 127.0.98.62:33161)
0927 12:45:26.076046 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0927 12:45:26.076050 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 12:45:26.076282 (+ 232us) server_negotiation.cc:407] Connection header received
0927 12:45:26.076386 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 12:45:26.076388 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 12:45:26.076541 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 12:45:26.076666 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 12:45:26.076877 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:26.077460 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:26.078557 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 12:45:26.078929 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 12:45:26.079034 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 12:45:26.079348 (+ 314us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 12:45:26.079750 (+ 402us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 12:45:26.079845 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 12:45:26.080172 (+ 327us) server_negotiation.cc:299] Negotiation successful
0927 12:45:26.080232 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":79,"threads_started":1}
I20260927 12:45:26.083509 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 23023
I20260927 12:45:26.090195 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 23098
I20260927 12:45:26.095887 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_Q2zcw/build/debug/bin/kudu with pid 23174
2026-09-27T12:45:26Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3111 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6197 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (285247 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST